Issues in installing a self written pandas based module on raspbian stretch

Hello,

I want to execute a script where as a requirement there is a self written module which on turn requires pandas. Despite different trials I met difficulties to make this work and I would be helpful if someone could look into the issue.

General setting

After a clean install of Python 3.7 I get the venv module with

sudo apt-get install python3-venv

The os is Raspbian stretch freshly installed. I only installed gfortran on top of it.

Sample code

Let us call the additional module 'fancy', with structure

- fancy
    > fancy
        *__init__.py
        *fancy_function.py
    > requirements.txt
    > setup.py

__init__.py is empty, while fancy_function.py is

import pandas as pd

def fancy_function():

    data = pd.DataFrame()
    print(type(data))

requirements.txt contains only the line pandas==0.23.3, while setup.py

from setuptools import setup, find_packages

setup(
name="fancy",
version='0.0.1',
packages=find_packages(),
install_requires=['pandas==0.23.3']
)

The content of the 'test.py' script where I want to import the module is

import fancy.fancy_function

print(fancy.fancy_function())

and now that everything is set up, let the show begin.

Trial 1: python3.7 setup.py install

Let's first create and venv with

python3.7 -m venv fancy_venv

Now to install the module we cd to its directory (after activating) and type

python3.7 setup.py install

we get...

    Processing pandas-0.23.3.tar.gz
Writing /tmp/easy_install-ftstskpz/pandas-0.23.3/setup.cfg
Running pandas-0.23.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-ftstskpz/pandas-0.23.3/egg-dist-tmp-c4nwafpi
Running from numpy source directory.
/tmp/easy_install-ftstskpz/pandas-0.23.3/temp/easy_install-zcrjk965/numpy-1.15.4/setup.py:378: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
  run_build = parse_setuppy_commands()
/tmp/easy_install-ftstskpz/pandas-0.23.3/temp/easy_install-zcrjk965/numpy-1.15.4/numpy/distutils/system_info.py:625: UserWarning: 
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  self.calc_info()
/tmp/easy_install-ftstskpz/pandas-0.23.3/temp/easy_install-zcrjk965/numpy-1.15.4/numpy/distutils/system_info.py:625: UserWarning: 
    Blas (http://www.netlib.org/blas/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [blas]) or by setting
    the BLAS environment variable.
  self.calc_info()
/tmp/easy_install-ftstskpz/pandas-0.23.3/temp/easy_install-zcrjk965/numpy-1.15.4/numpy/distutils/system_info.py:625: UserWarning: 
    Blas (http://www.netlib.org/blas/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [blas_src]) or by setting
    the BLAS_SRC environment variable.
  self.calc_info()
/bin/sh: 1: svnversion: not found
non-existing path in 'numpy/distutils': 'site.cfg'
/bin/sh: 1: svnversion: not found
/tmp/easy_install-ftstskpz/pandas-0.23.3/temp/easy_install-zcrjk965/numpy-1.15.4/numpy/distutils/system_info.py:625: UserWarning: 
    Lapack (http://www.netlib.org/lapack/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [lapack]) or by setting
    the LAPACK environment variable.
  self.calc_info()
/tmp/easy_install-ftstskpz/pandas-0.23.3/temp/easy_install-zcrjk965/numpy-1.15.4/numpy/distutils/system_info.py:625: UserWarning: 
    Lapack (http://www.netlib.org/lapack/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [lapack_src]) or by setting
    the LAPACK_SRC environment variable.
  self.calc_info()
/usr/local/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
  warnings.warn(msg)
_configtest.c:1:5: warning: conflicting types for built-in function ‘exp’
 int exp (void);
     ^~~
_configtest.o: In function `main':
/tmp/easy_install-ftstskpz/pandas-0.23.3/temp/easy_install-zcrjk965/numpy-1.15.4/_configtest.c:6: undefined reference to `exp'
collect2: error: ld returned 1 exit status
_configtest.o: In function `main':
/tmp/easy_install-ftstskpz/pandas-0.23.3/temp/easy_install-zcrjk965/numpy-1.15.4/_configtest.c:6: undefined reference to `exp'
collect2: error: ld returned 1 exit status
_configtest.c:1:5: warning: conflicting types for built-in function ‘exp’
 int exp (void);
     ^~~
_configtest.c:1:24: fatal error: sys/endian.h: No such file or directory
 #include <sys/endian.h>
                        ^
compilation terminated.
_configtest.c:1:24: fatal error: sys/endian.h: No such file or directory
 #include <sys/endian.h>

a little bit more and

#define HAVE_CREALL 1
#define HAVE_CSINL 1
#define HAVE_CSINHL 1
#define HAVE_CSQRTL 1
#define HAVE_CTANL 1
#define HAVE_CTANHL 1
#define NPY_RESTRICT restrict
#define NPY_RELAXED_STRIDES_CHECKING 1
#define HAVE_LDOUBLE_IEEE_DOUBLE_LE 1
#define NPY_PY3K 1
#ifndef __cplusplus
/* #undef inline */
#endif

#ifndef _NPY_NPY_CONFIG_H_
#error config.h should never be included directly, include npy_config.h instead
#endif

EOF

... and when you think it should be over ...

_configtest.c:1:5: warning: conflicting types for built-in function ‘exp’
 int exp (void);
     ^~~
_configtest.o: In function `main':
/tmp/easy_install-ftstskpz/pandas-0.23.3/temp/easy_install-zcrjk965/numpy-1.15.4/_configtest.c:6: undefined reference to `exp'
collect2: error: ld returned 1 exit status
_configtest.o: In function `main':
/tmp/easy_install-ftstskpz/pandas-0.23.3/temp/easy_install-zcrjk965/numpy-1.15.4/_configtest.c:6: undefined reference to `exp'
collect2: error: ld returned 1 exit status
_configtest.c:1:5: warning: conflicting types for built-in function ‘exp’
 int exp (void);
     ^~~
File: build/src.linux-armv7l-3.7/numpy/core/include/numpy/_numpyconfig.h
#define NPY_HAVE_ENDIAN_H 1
#define NPY_SIZEOF_SHORT SIZEOF_SHORT
#define NPY_SIZEOF_INT SIZEOF_INT
#define NPY_SIZEOF_LONG SIZEOF_LONG
#define NPY_SIZEOF_FLOAT 4
#define NPY_SIZEOF_COMPLEX_FLOAT 8
#define NPY_SIZEOF_DOUBLE 8
#define NPY_SIZEOF_COMPLEX_DOUBLE 16
#define NPY_SIZEOF_LONGDOUBLE 8
#define NPY_SIZEOF_COMPLEX_LONGDOUBLE 16
#define NPY_SIZEOF_PY_INTPTR_T 4
#define NPY_SIZEOF_OFF_T 8
#define NPY_SIZEOF_PY_LONG_LONG 8
#define NPY_SIZEOF_LONGLONG 8
#define NPY_NO_SMP 0
#define NPY_HAVE_DECL_ISNAN
#define NPY_HAVE_DECL_ISINF
#define NPY_HAVE_DECL_ISFINITE
#define NPY_HAVE_DECL_SIGNBIT
#define NPY_USE_C99_COMPLEX 1
#define NPY_HAVE_COMPLEX_DOUBLE 1
#define NPY_HAVE_COMPLEX_FLOAT 1
#define NPY_HAVE_COMPLEX_LONG_DOUBLE 1
#define NPY_RELAXED_STRIDES_CHECKING 1
#define NPY_USE_C99_FORMATS 1
#define NPY_VISIBILITY_HIDDEN __attribute__((visibility("hidden")))
#define NPY_ABI_VERSION 0x01000009
#define NPY_API_VERSION 0x0000000C

#ifndef __STDC_FORMAT_MACROS
#define __STDC_FORMAT_MACROS 1
#endif

EOF
### Warning:  Using unoptimized lapack ###
### Warning:  Using unoptimized lapack ###
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '*.pyd' found anywhere in distribution
warning: no previously-included files matching '*.swp' found anywhere in distribution
warning: no previously-included files matching '*.bak' found anywhere in distribution
warning: no previously-included files matching '*~' found anywhere in distribution
In file included from numpy/core/src/npymath/npy_math.c:9:0:
numpy/core/src/npymath/npy_math_internal.h.src: In function ‘npy_modfl’:
numpy/core/src/npymath/npy_math_internal.h.src:490:21: warning: passing argument 2 of ‘modfl’ from incompatible pointer type [-Wincompatible-pointer-types]
     return modf@c@(x, iptr);
                     ^~~~
In file included from /usr/local/include/python3.7m/pyport.h:191:0,
                 from /usr/local/include/python3.7m/Python.h:53,
                 from numpy/core/src/npymath/npy_math_private.h:21,
                 from numpy/core/src/npymath/npy_math_internal.h.src:56,
                 from numpy/core/src/npymath/npy_math.c:9:
/usr/include/arm-linux-gnueabihf/bits/mathcalls.h:115:1: note: expected ‘long double *’ but argument is of type ‘npy_longdouble * {aka double *}’
 __MATHCALL (modf,, (_Mdouble_ __x, _Mdouble_ *__iptr)) __nonnull ((2));
 ^
In file included from numpy/core/include/numpy/npy_math.h:548:0,
                 from numpy/core/src/multiarray/compiled_base.c:9:
numpy/core/src/npymath/npy_math_internal.h.src: In function ‘npy_modfl’:
numpy/core/src/npymath/npy_math_internal.h.src:490:21: warning: passing argument 2 of ‘modfl’ from incompatible pointer type [-Wincompatible-pointer-types]
     return modf@c@(x, iptr);
                     ^~~~
In file included from /usr/local/include/python3.7m/pyport.h:191:0,
                 from /usr/local/include/python3.7m/Python.h:53,
                 from numpy/core/src/multiarray/compiled_base.c:2:
/usr/include/arm-linux-gnueabihf/bits/mathcalls.h:115:1: note: expected ‘long double *’ but argument is of type ‘npy_longdouble * {aka double *}’
 __MATHCALL (modf,, (_Mdouble_ __x, _Mdouble_ *__iptr)) __nonnull ((2));
 ^

At this point I kill the process. Fail.

Second attempt: pip install .

Let us trash the old venv and create it from scratch. This time I cd to the fancy folder and type:

pip install .

This time goes much faster:

Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Processing /home/pi/production/fancy
Collecting pandas==0.23.3 (from fancy==0.0.1)
  Using cached https://files.pythonhosted.org/packages/3e/56/82c4d4c049294f87ebd05b65fdcbc9ed68bd23fb0a7e4469caf9a75d199f/pandas-0.23.3.tar.gz

and it does no longer give signs of life. Ctrl-c. Fail.

Third attempt: remembering to upgrade pip

Hoping that pip 10.0.1 might be the cause of my issues I first run

pip install --upgrade pip

Now I have pip 18.1 and I run

pip install .

The same. Fail.

Fourth attempt: use python 3.5 and pip install .

Basically like point 3 and for, but the venv is made in python3.5. The I run (pip 10.0.1):

pip install .

and we get ...

Processing /home/pi/production/fancy
Collecting pandas==0.23.3 (from fancy==0.0.1)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Using cached https://www.piwheels.org/simple/pandas/pandas-0.23.3-cp35-cp35m-linux_armv7l.whl
Collecting python-dateutil>=2.5.0 (from pandas==0.23.3->fancy==0.0.1)
  Using cached https://files.pythonhosted.org/packages/74/68/d87d9b36af36f44254a8d512cbfc48369103a3b9e474be9bdfe536abfc45/python_dateutil-2.7.5-py2.py3-none-any.whl
Collecting numpy>=1.9.0 (from pandas==0.23.3->fancy==0.0.1)
  Using cached https://www.piwheels.org/simple/numpy/numpy-1.15.4-cp35-cp35m-linux_armv7l.whl
Collecting pytz>=2011k (from pandas==0.23.3->fancy==0.0.1)
  Using cached https://files.pythonhosted.org/packages/f8/0e/2365ddc010afb3d79147f1dd544e5ee24bf4ece58ab99b16fbb465ce6dc0/pytz-2018.7-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil>=2.5.0->pandas==0.23.3->fancy==0.0.1)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, numpy, pytz, pandas, fancy
  Running setup.py install for fancy ... done
Successfully installed fancy-0.0.1 numpy-1.15.4 pandas-0.23.3 python-dateutil-2.7.5 pytz-2018.7 six-1.12.0

Joy, success!!! Why am I asking for help in stack if this works? Let us run my compicated test.py

python3.5 test.py

bitte:

Traceback (most recent call last):
  File "test.py", line 1, in <module>
    import fancy.fancy_function
  File "/home/pi/production/env/fancy_venv_1/lib/python3.5/site-packages/fancy/fancy_function.py", line 1, in <module>
    import pandas as pd
  File "/home/pi/production/env/fancy_venv_1/lib/python3.5/site-packages/pandas/__init__.py", line 19, in <module>
    "Missing required dependencies {0}".format(missing_dependencies))
ImportError: Missing required dependencies ['numpy']

White flag. Any idea to solve this is welcome.

Kind regards

embè