Python setup.py 簡易研究

最近在公司開發 Python 專案時，有一些需求

1. 保護 Source code 的方法。網路上其實可以找到很多方法，像是代碼混淆、加密、打包成一個可執行檔以及利用 Cython 將 Python 檔案轉換為 C，然後編繹成 Library 檔案 (Windows 為 pyd 檔案 , Linux 為 so 檔案)。最終採用了最後兩種方法
2. 大多的專案最終都會有重覆的程式，所以想寫一個共用 Library 供開發使用

上述兩點都可以使用 Python Setup.py 來實現，所以要先了解 Setup.py 怎麼撰寫

一、先來看看 Setup.py Spec 吧

想要了解 Setup.py 是什麼，或是如何寫出一個 Setup.py？
我會請你直接 Google (呵呵)

原因無它，由於 Python 文件檔案做得很好，沒有理由不利用它

• Writing the Setup Script
• List Classifiers

這裡挑幾個常用到的名詞做簡單的說明

• name : 一般填寫 package 的名字即可
• version : 版本
• description : 基本描述
• logn_description : 更加詳細的描述
• author : 作者
• author_email : email
• url : 項目的網站
• license : 授權的方式
• keywords : 關鍵字，可作為分類用
• packages : 告訴 Distutils 需要處理哪些 package (有包含 __init__.py 的資料夾)
• package_dir : 告訴 Distutils 哪些 package 所對應的相對路徑。比如說 package_dir={'foo', 'lib'}，表示路徑最終為 'lib/foo'
• ext_modules : 是一個包含 Extension 的列表
• ext_package : 定義 Extension 的相對路徑
• requires : 定義依賴哪些 package or module
• provides : 定義可以為哪些 module 提供依賴 (Dependency)
• scripts : 指定一個可以從 command line 執行的 python 檔案，在安裝時指定 --install-script
• package_date : 通常用於相關的數據文件或類似於 readme 的文件。比如說 package_data={'':[*.txt'], 'mypkg':['data/*.dat']}，表示包含所有資料夾的 txt 檔案和 mypkg/data中所有 dat 檔案
• data_files : 指定其它的一些文件，如 config 檔案。比如說 data_files=[(bitmaps', ['bm/b1.gif', 'bm/b2.gif'])]，表示 'b1.gif' 和 'b1.gif' 將會放在 bitmaps 資料夾中。
• 另一種方式是寫一個 MANIFEST.in template，將需要包含在分發包的文件寫在裡面

參考上面寫一個簡單的 Setup.py，大概是長成這樣

from distutils.core import setup
 
setup(
    name="text",
    version="1.0",
    author="YOUR_NAME"
    author_email="YOUR_NAME@email",
    url="http://www.test.org",
    packages=['test'],
)

二、Setup.py 有哪些指令可以用

除了上述的方法，其實也可以用 help 指令去了解 Setup.py

$> python setup.py --help
 
Common commands: (see '--help-commands' for more)
 
  setup.py build      will build the package underneath 'build/'
  setup.py install    will install the package
 
Global options:
  --verbose (-v)      run verbosely (default)
  --quiet (-q)        run quietly (turns verbosity off)
  --dry-run (-n)      don't actually do anything
  --help (-h)         show detailed help message
  --no-user-cfg       ignore pydistutils.cfg in your home directory
  --command-packages  list of packages that provide distutils commands
 
Information display options (just display information, ignore any commands)
  --help-commands     list all available commands
  --name              print package name
  --version (-V)      print package version
  --fullname          print -
  --author            print the author's name
  --author-email      print the author's email address
  --maintainer        print the maintainer's name
  --maintainer-email  print the maintainer's email address
  --contact           print the maintainer's name if known, else the author's
  --contact-email     print the maintainer's email address if known, else the
                      author's
  --url               print the URL for this package
  --license           print the license of the package
  --licence           alias for --license
  --description       print the package description
  --long-description  print the long package description
  --platforms         print the list of platforms
  --classifiers       print the list of classifiers
  --keywords          print the list of keywords
  --provides          print the list of packages/modules provided
  --requires          print the list of packages/modules required
  --obsoletes         print the list of packages/modules made obsolete
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

試試 Commands help

$> python setup.py --help-command
 
Standard commands:
  build             build everything needed to install
  build_py          "build" pure Python modules (copy to build directory)
  build_ext         build C/C++ and Cython extensions (compile/link to build directory)
  build_clib        build C/C++ libraries used by Python extensions
  build_scripts     "build" scripts (copy and fixup #! line)
  clean             clean up temporary files from 'build' command
  install           install everything from build directory
  install_lib       install all Python modules (extensions and pure Python)
  install_headers   install C/C++ header files
  install_scripts   install scripts (Python or otherwise)
  install_data      install data files
  sdist             create a source distribution (tarball, zip file, etc.)
  register          register the distribution with the Python package index
  bdist             create a built (binary) distribution
  bdist_dumb        create a "dumb" built distribution
  bdist_rpm         create an RPM distribution
  bdist_wininst     create an executable installer for MS Windows
  upload            upload binary package to PyPI
  check             perform some checks on the package
 
Extra commands:
  saveopts          save supplied options to setup.cfg or other config file
  develop           install package in 'development mode'
  upload_docs       Upload documentation to PyPI
  test              run unit tests after in-place build
  setopt            set an option in setup.cfg or another config file
  install_egg_info  Install an .egg-info directory for the package
  rotate            delete older distributions, keeping N newest files
  cleanall          clean up temporary files from 'build' command
  bdist_wheel       create a wheel distribution
  egg_info          create a distribution's .egg-info directory
  alias             define a shortcut to invoke one or more commands
  easy_install      Find/get/install Python packages
  bdist_egg         create an "egg" distribution
 
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

這裡舉例一些常用的指令

python setup.py build            # 僅編繹不安裝
python setup.py install          # 安裝到 python 安裝目錄的 lib
python setup.py sdist            # 產生成壓縮檔案 (zip/tar.gz)。Support pip
 
python setup.py bdist --formats=zip
                                 # 等同於 python setup.py sdist
python setup.py bdist_egg        # 產生成一個二進制分發版本，經常用來替代 bdist。Support easy_install
python setup.py bdist_wininst    # 生成 NT 平台安裝檔案 (.exe)
python setup.py bdist_rpm        # 生成 rpm 檔案
 
python setup.py develop          # 編繹並且在適當的位置安裝，然後增加一個鏈結到 python site-packages 中
python setup.py develop -u       # 反安裝

三、利用 Cython 將 Python 檔案轉成 C 檔案

Cython 官網和文件在這裡，因為我沒有看完，就不多說了

1. 安裝方法並不困難
   

   pip install Cython

2. 將原本的 Python 檔案，重新命名副檔名為 *.pyx
3. 簡單的 Setup.py 可以長這樣
   

   from distutils.core import setup
   from Cython.Build import cythonize
    
   setup(
     name = 'example 1',
     ext_modules = cythonize("*.pyx"),
   )

4. 還有使用 Extension 的寫法，像這樣
   

   from distutils.core import setup
   from distutils.extension import Extension
   from Cython.Build import cythonize
    
   ext_modules=[
       Extension(
           "example",
           sources=["example.pyx"]
       )
   ]
    
   setup(
     name = "example 2",
     ext_modules = cythonize(ext_modules)
   )

5. 執行下列指令，Cython 就會將 *pyx 轉換為 *.c ，經由 gcc 編譯之後就可以得到 library 檔案囉
   

   python setup.py build_ext --inplace

四、來個 Setup.py 範例吧

檔案的架構長這樣

setup_example/
├── Makefile
├── setup.py
└── src
    ├── Hello
    │   ├── Hello.py
    │   └── __init__.py
    ├── Hello_so
    │   ├── Hello_so.pyx
    │   └── __init__.py
    └── __init__.py

4.1 Source code

• Hello.__init__.py
  

  from __future__ import absolute_import
   
  from .Hello import hello_world
   

• Hello.Hello.py
  

  def hello_world():
      print("PY SAY: Hello!!")
   

• Hello_so.__init__.py
  

  from __future__ import absolute_import
   
  from .Hello_so import hello_world
   

• Hello_so.Hello_so.pyx
  

  def hello_world():
      print("SO SAY: Hello!!")
   

• setup.py
  

  # coding: utf-8
  import glob
  import os
  from distutils.command.clean import clean
  from distutils.core import setup
   
  from setuptools import find_packages
  from setuptools.extension import Extension
   
  try:
      from Cython.Distutils import build_ext
  except (ImportError, ImportWarning):
      print("Import Cython failed. Is Cython not installed?")
      raise
   
  Project_Name = 'setup_example'
  Project_PKG_Path = 'src'
   
  LONGDESC = """
  Example Python Setup.py
  =======================
   
  How to build
  ============
   
  $> python setup.py bdist_egg                        # generate a build .egg file
  $> python setup.py bdist_egg --exclude-source-files # generate a build .egg file without source files
  $> python setup.py sdist                            # generate a build source distro
  $> python setup.py sdist bdist_egg                  # generate both
   
  How to install
  ==============
   
  $> python setup.py install
  $> pip install dist/'generated-egg'
   
  How to uninstall
  ================
   
  $> pip uninstall {name}
   
  """.format(name=Project_Name)
   
  Requires = [
      'cython',
  ]
   
   
  def _searching_dir(current_dir, extension, files=list()):
      """ Create a list of all files to be compiled """
      for file_name in os.listdir(current_dir):
          file_path = os.path.join(current_dir, file_name)
          if os.path.isfile(file_path) and file_path.endswith(extension):
              files.append(file_path)
          elif os.path.isdir(file_path):
              _searching_dir(file_path, extension, files)
   
      return files
   
   
  def _make_extension(ext_path, only_filename=False, includes=list()):
      """Generate an Extension object from its dotted name
   
      :param ext_path:
      :param only_filename:
      :param includes: Including path
      :return:
      """
      if only_filename is True:
          ext_path = os.path.basename(ext_path)
   
      return Extension(
          (ext_path.replace(os.path.sep, '.'))[:-4],
          [ext_path],
          include_dirs=includes + ['.'],  # adding the '.' to include_dirs is CRUCIAL!!
          # extra_compile_args = ["-O3", "-Wall"],
          # extra_link_args = ['-g'],
          # libraries = ["dv",],
      )
   
   
  def _purge(pattern):
      for path in glob.glob(pattern):
          if os.path.exists(path):
              os.remove(path)
   
   
  class CleanAll(clean):
      def run(self):
          # Clean build
          clean.run(self)
   
          # Clean *.c, *.so generated by pyx file
          for file_path in pyx_modules:
              _purge(file_path[:-4] + "*.c")
              _purge(file_path[:-4] + '*.so')
   
   
  if __name__ == '__main__':
   
      # build up the set of Extension objects
      ext_modules = []
      pyx_modules = _searching_dir(Project_PKG_Path, ".pyx", [])
      for name in pyx_modules:
          ext_modules.append(_make_extension(name))
   
      setup(
          name=Project_Name,
          version="0.0",
          requires=Requires,
          cmdclass={'build_ext': build_ext, 'cleanall': CleanAll},
          description='OnWord Security Common Python Utilities',
          long_description=LONGDESC,
          author='Your Name',
          author_email='YOUR_NAME@example.com',
          url='https://www.example.com/',
          license='Private',
          classifiers=[
              'Intended Audience :: Developers',
              'Operating System :: OS Independent',
              'Programming Language :: Python',
              'Programming Language :: Python :: 2.7',
              'Topic :: Software Development :: Libraries',
              'Topic :: Utilities',
          ],
          keywords='example',
          packages=find_packages(exclude=[], ),
          package_dir={'src': 'src'},
          package_data={'src': ['*.*']},
          zip_safe=False,
          ext_modules=ext_modules,
      )
   

• Makefile
  

  # Makefile
  #
  # @build
  # 1. make, or
  # 2. make all
  #
  # @install
  # 1. make install
  #
  # @clean all
  # 1. make clean
  #
  # @Change python or pyinstaller path
  #
  PYTHON:=python
   
  all:
   $(PYTHON) setup.py build_ext --inplace
   
  .PHONY: all
   
  build-clean:
   $(PYTHON) setup.py cleanall
   rm -rf MANIFEST build dist *.egg-info
   
  .PHONY: build-clean
   
  sdist: all
   $(PYTHON) setup.py sdist
   
  .PHONY: sdist
   
  install: all
   $(PYTHON) setup.py install
   
  .PHONY: install
   
  clean: build-clean
   find . -type f -name "*.py[co]" -delete
   find . -type d -name "__pycache__" -delete
   
  .PHONY: clean

4.2 安裝到 python lib 中

$> make install

如果沒出現錯誤訊息的話，那麼應該會順利地編譯出 so 檔案

可以用  pip list 看看已安裝的 packages

$> pip list
 
appdirs (1.4.0)
Cython (0.25.2)
packaging (16.8)
pip (9.0.1)
pyparsing (2.1.10)
setup-example (0.0)
setuptools (34.2.0)
six (1.10.0)
wheel (0.29.0)

有看到 setup-example (0.0)，接下來就是試用看看囉

4.3 用看看效果如何

from src.Hello import hello_world
from src.Hello_so import hello_world as hello_world_so
 
hello_world()
hello_world_so()
 

結果畫面為

最後試著用 pyinstaller，so 檔也可以成功地打包
順帶一提，反安裝的方法可以用

pip uninstall setup-example

5. Q&A

• 出現 error: each element of 'ext_modules' option must be an Extension instance or 2-tuple
  解決方法 : 雖然目前還是不明白為什麼，改用 setuptools extension 就沒問題了
  

  +   from setuptools.extension import Extension
  -   from distutils.extension import Extension