NumPy ndarray Vs. Python Lists

NumPy is a package for scientific computing and used to overcome Python’s limitation of slow processing time for multidimensional arrays via lists. In other words, it is an extension to Python to use multidimensional arrays as native objects. NumPy arrays are especially written keeping this multi-dimension use case in mind and hence, provide better performance in terms of both speed and memory. 

Why is it more efficient?

Python’s lists do not have to be homogeneous. They can have a string element, an integer and a float. To create a structure to support all types, CPython implements it like so.

Here, PyObject and PyTypeObject store methods, i/o and subclassing attributes. 

“`typedef struct _object {
    _PyObject_HEAD_EXTRA
    Py_ssize_t ob_refcnt;
    struct _typeobject *ob_type;
} PyObject;

typedef struct _typeobject {
    PyObject_VAR_HEAD
    const char *tp_name; /* For printing, in format “<module>.<name>” */
    Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */

    /* Methods to implement standard operations */

    destructor tp_dealloc;
    Py_ssize_t tp_vectorcall_offset;
    getattrfunc tp_getattr;
    setattrfunc tp_setattr;
    PyAsyncMethods *tp_as_async; /* formerly known as tp_compare (Python 2)
                                    or tp_reserved (Python 3) */
    reprfunc tp_repr;

    /* Method suites for standard classes */

    PyNumberMethods *tp_as_number;
    PySequenceMethods *tp_as_sequence;
    PyMappingMethods *tp_as_mapping;

    /* More standard operations (here for binary compatibility) */

    hashfunc tp_hash;
    ternaryfunc tp_call;
    reprfunc tp_str;
    getattrofunc tp_getattro;
    setattrofunc tp_setattro;

    /* Functions to access object as input/output buffer */
    PyBufferProcs *tp_as_buffer;

    /* Flags to define presence of optional/expanded features */
    unsigned long tp_flags;

    const char *tp_doc; /* Documentation string */

    /* Assigned meaning in release 2.0 */
    /* call function for all accessible objects */
    traverseproc tp_traverse;

    /* delete references to contained objects */
    inquiry tp_clear;

    /* Assigned meaning in release 2.1 */
    /* rich comparisons */
    richcmpfunc tp_richcompare;

    /* weak reference enabler */
    Py_ssize_t tp_weaklistoffset;

    /* Iterators */
    getiterfunc tp_iter;
    iternextfunc tp_iternext;

    /* Attribute descriptor and subclassing stuff */
    struct PyMethodDef *tp_methods;
    struct PyMemberDef *tp_members;
    struct PyGetSetDef *tp_getset;
    struct _typeobject *tp_base;
    PyObject *tp_dict;
    descrgetfunc tp_descr_get;
    descrsetfunc tp_descr_set;
    Py_ssize_t tp_dictoffset;
    initproc tp_init;
    allocfunc tp_alloc;
    newfunc tp_new;
    freefunc tp_free; /* Low-level free-memory routine */
    inquiry tp_is_gc; /* For PyObject_IS_GC */
    PyObject *tp_bases;
    PyObject *tp_mro; /* method resolution order */
    PyObject *tp_cache;
    PyObject *tp_subclasses;
    PyObject *tp_weaklist;
    destructor tp_del;

    /* Type attribute cache version tag. Added in version 2.6 */
    unsigned int tp_version_tag;

    destructor tp_finalize;
    vectorcallfunc tp_vectorcall;

#ifdef COUNT_ALLOCS
    /* these must be last and never explicitly initialized */
    Py_ssize_t tp_allocs;
    Py_ssize_t tp_frees;
    Py_ssize_t tp_maxalloc;
    struct _typeobject *tp_prev;
    struct _typeobject *tp_next;
#endif
} PyTypeObject;“`However, NumPy’s array uses PyArrayObject defined considering the type of operations that it would deal with.

typedef PyArrayObject_fields PyArrayObject;

typedef struct tagPyArrayObject_fields {
    PyObject_HEAD
    /* Pointer to the raw data buffer */
    char *data;
    /* The number of dimensions, also called ‘ndim’ */
    int nd;
    /* The size in each dimension, also called ‘shape’ */
    npy_intp *dimensions;
    /*
     * Number of bytes to jump to get to the
     * next element in each dimension
     */
    npy_intp *strides;
    /*
     * This object is decref’d upon
     * deletion of array. Except in the
     * case of WRITEBACKIFCOPY which has
     * special handling.
     *
     * For views it points to the original
     * array, collapsed so no chains of
     * views occur.
     *
     * For creation from buffer object it
     * points to an object that should be
     * decref’d on deletion
     *
     * For WRITEBACKIFCOPY flag this is an
     * array to-be-updated upon calling
     * PyArray_ResolveWritebackIfCopy
     */
    PyObject *base;
    /* Pointer to type structure */
    PyArray_Descr *descr;
    /* Flags describing array — see below */
    int flags;
    /* For weak references */
    PyObject *weakreflist;
} PyArrayObject_fields;

The source for above definitions can be found on GitHub: https://github.com/numpy/numpy/blob/master/numpy/core/include/numpy/ndarraytypes.h

The element size is fixed for each ndarray and can be accessed using:

static NPY_INLINE npy_intp
PyArray_ITEMSIZE(const PyArrayObject *arr)
{
    return ((PyArrayObject_fields *)arr)->descr->elsize;
}

Similarly, there are other macros and definitions for PyArray in the above link and can be used to check how getters and setters work. 

Official Scipy documentation for PyArrayObject: https://docs.scipy.org/doc/numpy/reference/c-api.types-and-structures.html#c.PyArrayObject

Article Contributed By:
Chandrika Mutalik
Software Engineer, Pluribus Networks

Leave a Reply

Your email address will not be published. Required fields are marked *