This commit is contained in:
2025-09-07 22:09:54 +02:00
parent e1b817252c
commit 2fc0d000b6
7796 changed files with 2159515 additions and 933 deletions

View File

@ -0,0 +1,135 @@
.. -*- rest -*-
==================================================
API changes in the new masked array implementation
==================================================
Masked arrays are subclasses of ndarray
---------------------------------------
Contrary to the original implementation, masked arrays are now regular
ndarrays::
>>> x = masked_array([1,2,3],mask=[0,0,1])
>>> print isinstance(x, numpy.ndarray)
True
``_data`` returns a view of the masked array
--------------------------------------------
Masked arrays are composed of a ``_data`` part and a ``_mask``. Accessing the
``_data`` part will return a regular ndarray or any of its subclass, depending
on the initial data::
>>> x = masked_array(numpy.matrix([[1,2],[3,4]]),mask=[[0,0],[0,1]])
>>> print x._data
[[1 2]
[3 4]]
>>> print type(x._data)
<class 'numpy.matrixlib.defmatrix.matrix'>
In practice, ``_data`` is implemented as a property, not as an attribute.
Therefore, you cannot access it directly, and some simple tests such as the
following one will fail::
>>>x._data is x._data
False
``filled(x)`` can return a subclass of ndarray
----------------------------------------------
The function ``filled(a)`` returns an array of the same type as ``a._data``::
>>> x = masked_array(numpy.matrix([[1,2],[3,4]]),mask=[[0,0],[0,1]])
>>> y = filled(x)
>>> print type(y)
<class 'numpy.matrixlib.defmatrix.matrix'>
>>> print y
matrix([[ 1, 2],
[ 3, 999999]])
``put``, ``putmask`` behave like their ndarray counterparts
-----------------------------------------------------------
Previously, ``putmask`` was used like this::
mask = [False,True,True]
x = array([1,4,7],mask=mask)
putmask(x,mask,[3])
which translated to::
x[~mask] = [3]
(Note that a ``True``-value in a mask suppresses a value.)
In other words, the mask had the same length as ``x``, whereas
``values`` had ``sum(~mask)`` elements.
Now, the behaviour is similar to that of ``ndarray.putmask``, where
the mask and the values are both the same length as ``x``, i.e.
::
putmask(x,mask,[3,0,0])
``fill_value`` is a property
----------------------------
``fill_value`` is no longer a method, but a property::
>>> print x.fill_value
999999
``cumsum`` and ``cumprod`` ignore missing values
------------------------------------------------
Missing values are assumed to be the identity element, i.e. 0 for
``cumsum`` and 1 for ``cumprod``::
>>> x = N.ma.array([1,2,3,4],mask=[False,True,False,False])
>>> print x
[1 -- 3 4]
>>> print x.cumsum()
[1 -- 4 8]
>> print x.cumprod()
[1 -- 3 12]
``bool(x)`` raises a ValueError
-------------------------------
Masked arrays now behave like regular ``ndarrays``, in that they cannot be
converted to booleans:
::
>>> x = N.ma.array([1,2,3])
>>> bool(x)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
==================================
New features (non exhaustive list)
==================================
``mr_``
-------
``mr_`` mimics the behavior of ``r_`` for masked arrays::
>>> np.ma.mr_[3,4,5]
masked_array(data = [3 4 5],
mask = False,
fill_value=999999)
``anom``
--------
The ``anom`` method returns the deviations from the average (anomalies).

View File

@ -0,0 +1,24 @@
* Copyright (c) 2006, University of Georgia and Pierre G.F. Gerard-Marchant
* All rights reserved.
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of the University of Georgia nor the
* names of its contributors may be used to endorse or promote products
* derived from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR ANY
* DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
* (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
* ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

View File

@ -0,0 +1,236 @@
==================================
A guide to masked arrays in NumPy
==================================
.. Contents::
See http://www.scipy.org/scipy/numpy/wiki/MaskedArray (dead link)
for updates of this document.
History
-------
As a regular user of MaskedArray, I (Pierre G.F. Gerard-Marchant) became
increasingly frustrated with the subclassing of masked arrays (even if
I can only blame my inexperience). I needed to develop a class of arrays
that could store some additional information along with numerical values,
while keeping the possibility for missing data (picture storing a series
of dates along with measurements, what would later become the `TimeSeries
Scikit <http://projects.scipy.org/scipy/scikits/wiki/TimeSeries>`__
(dead link).
I started to implement such a class, but then quickly realized that
any additional information disappeared when processing these subarrays
(for example, adding a constant value to a subarray would erase its
dates). I ended up writing the equivalent of *numpy.core.ma* for my
particular class, ufuncs included. Everything went fine until I needed to
subclass my new class, when more problems showed up: some attributes of
the new subclass were lost during processing. I identified the culprit as
MaskedArray, which returns masked ndarrays when I expected masked
arrays of my class. I was preparing myself to rewrite *numpy.core.ma*
when I forced myself to learn how to subclass ndarrays. As I became more
familiar with the *__new__* and *__array_finalize__* methods,
I started to wonder why masked arrays were objects, and not ndarrays,
and whether it wouldn't be more convenient for subclassing if they did
behave like regular ndarrays.
The new *maskedarray* is what I eventually come up with. The
main differences with the initial *numpy.core.ma* package are
that MaskedArray is now a subclass of *ndarray* and that the
*_data* section can now be any subclass of *ndarray*. Apart from a
couple of issues listed below, the behavior of the new MaskedArray
class reproduces the old one. Initially the *maskedarray*
implementation was marginally slower than *numpy.ma* in some areas,
but work is underway to speed it up; the expectation is that it can be
made substantially faster than the present *numpy.ma*.
Note that if the subclass has some special methods and
attributes, they are not propagated to the masked version:
this would require a modification of the *__getattribute__*
method (first trying *ndarray.__getattribute__*, then trying
*self._data.__getattribute__* if an exception is raised in the first
place), which really slows things down.
Main differences
----------------
* The *_data* part of the masked array can be any subclass of ndarray (but not recarray, cf below).
* *fill_value* is now a property, not a function.
* in the majority of cases, the mask is forced to *nomask* when no value is actually masked. A notable exception is when a masked array (with no masked values) has just been unpickled.
* I got rid of the *share_mask* flag, I never understood its purpose.
* *put*, *putmask* and *take* now mimic the ndarray methods, to avoid unpleasant surprises. Moreover, *put* and *putmask* both update the mask when needed. * if *a* is a masked array, *bool(a)* raises a *ValueError*, as it does with ndarrays.
* in the same way, the comparison of two masked arrays is a masked array, not a boolean
* *filled(a)* returns an array of the same subclass as *a._data*, and no test is performed on whether it is contiguous or not.
* the mask is always printed, even if it's *nomask*, which makes things easy (for me at least) to remember that a masked array is used.
* *cumsum* works as if the *_data* array was filled with 0. The mask is preserved, but not updated.
* *cumprod* works as if the *_data* array was filled with 1. The mask is preserved, but not updated.
New features
------------
This list is non-exhaustive...
* the *mr_* function mimics *r_* for masked arrays.
* the *anom* method returns the anomalies (deviations from the average)
Using the new package with numpy.core.ma
----------------------------------------
I tried to make sure that the new package can understand old masked
arrays. Unfortunately, there's no upward compatibility.
For example:
>>> import numpy.core.ma as old_ma
>>> import maskedarray as new_ma
>>> x = old_ma.array([1,2,3,4,5], mask=[0,0,1,0,0])
>>> x
array(data =
[ 1 2 999999 4 5],
mask =
[False False True False False],
fill_value=999999)
>>> y = new_ma.array([1,2,3,4,5], mask=[0,0,1,0,0])
>>> y
array(data = [1 2 -- 4 5],
mask = [False False True False False],
fill_value=999999)
>>> x==y
array(data =
[True True True True True],
mask =
[False False True False False],
fill_value=?)
>>> old_ma.getmask(x) == new_ma.getmask(x)
array([True, True, True, True, True])
>>> old_ma.getmask(y) == new_ma.getmask(y)
array([True, True, False, True, True])
>>> old_ma.getmask(y)
False
Using maskedarray with matplotlib
---------------------------------
Starting with matplotlib 0.91.2, the masked array importing will work with
the maskedarray branch) as well as with earlier versions.
By default matplotlib still uses numpy.ma, but there is an rcParams setting
that you can use to select maskedarray instead. In the matplotlibrc file
you will find::
#maskedarray : False # True to use external maskedarray module
# instead of numpy.ma; this is a temporary #
setting for testing maskedarray.
Uncomment and set to True to select maskedarray everywhere.
Alternatively, you can test a script with maskedarray by using a
command-line option, e.g.::
python simple_plot.py --maskedarray
Masked records
--------------
Like *numpy.ma.core*, the *ndarray*-based implementation
of MaskedArray is limited when working with records: you can
mask any record of the array, but not a field in a record. If you
need this feature, you may want to give the *mrecords* package
a try (available in the *maskedarray* directory in the scipy
sandbox). This module defines a new class, *MaskedRecord*. An
instance of this class accepts a *recarray* as data, and uses two
masks: the *fieldmask* has as many entries as records in the array,
each entry with the same fields as a record, but of boolean types:
they indicate whether the field is masked or not; a record entry
is flagged as masked in the *mask* array if all the fields are
masked. A few examples in the file should give you an idea of what
can be done. Note that *mrecords* is still experimental...
Optimizing maskedarray
----------------------
Should masked arrays be filled before processing or not?
--------------------------------------------------------
In the current implementation, most operations on masked arrays involve
the following steps:
* the input arrays are filled
* the operation is performed on the filled arrays
* the mask is set for the results, from the combination of the input masks and the mask corresponding to the domain of the operation.
For example, consider the division of two masked arrays::
import numpy
import maskedarray as ma
x = ma.array([1,2,3,4],mask=[1,0,0,0], dtype=numpy.float64)
y = ma.array([-1,0,1,2], mask=[0,0,0,1], dtype=numpy.float64)
The division of x by y is then computed as::
d1 = x.filled(0) # d1 = array([0., 2., 3., 4.])
d2 = y.filled(1) # array([-1., 0., 1., 1.])
m = ma.mask_or(ma.getmask(x), ma.getmask(y)) # m =
array([True,False,False,True])
dm = ma.divide.domain(d1,d2) # array([False, True, False, False])
result = (d1/d2).view(MaskedArray) # masked_array([-0. inf, 3., 4.])
result._mask = logical_or(m, dm)
Note that a division by zero takes place. To avoid it, we can consider
to fill the input arrays, taking the domain mask into account, so that::
d1 = x._data.copy() # d1 = array([1., 2., 3., 4.])
d2 = y._data.copy() # array([-1., 0., 1., 2.])
dm = ma.divide.domain(d1,d2) # array([False, True, False, False])
numpy.putmask(d2, dm, 1) # d2 = array([-1., 1., 1., 2.])
m = ma.mask_or(ma.getmask(x), ma.getmask(y)) # m =
array([True,False,False,True])
result = (d1/d2).view(MaskedArray) # masked_array([-1. 0., 3., 2.])
result._mask = logical_or(m, dm)
Note that the *.copy()* is required to avoid updating the inputs with
*putmask*. The *.filled()* method also involves a *.copy()*.
A third possibility consists in avoid filling the arrays::
d1 = x._data # d1 = array([1., 2., 3., 4.])
d2 = y._data # array([-1., 0., 1., 2.])
dm = ma.divide.domain(d1,d2) # array([False, True, False, False])
m = ma.mask_or(ma.getmask(x), ma.getmask(y)) # m =
array([True,False,False,True])
result = (d1/d2).view(MaskedArray) # masked_array([-1. inf, 3., 2.])
result._mask = logical_or(m, dm)
Note that here again the division by zero takes place.
A quick benchmark gives the following results:
* *numpy.ma.divide* : 2.69 ms per loop
* classical division : 2.21 ms per loop
* division w/ prefilling : 2.34 ms per loop
* division w/o filling : 1.55 ms per loop
So, is it worth filling the arrays beforehand ? Yes, if we are interested
in avoiding floating-point exceptions that may fill the result with infs
and nans. No, if we are only interested into speed...
Thanks
------
I'd like to thank Paul Dubois, Travis Oliphant and Sasha for the
original masked array package: without you, I would never have started
that (it might be argued that I shouldn't have anyway, but that's
another story...). I also wish to extend these thanks to Reggie Dugard
and Eric Firing for their suggestions and numerous improvements.
Revision notes
--------------
* 08/25/2007 : Creation of this page
* 01/23/2007 : The package has been moved to the SciPy sandbox, and is regularly updated: please check out your SVN version!

View File

@ -0,0 +1,53 @@
"""
=============
Masked Arrays
=============
Arrays sometimes contain invalid or missing data. When doing operations
on such arrays, we wish to suppress invalid values, which is the purpose masked
arrays fulfill (an example of typical use is given below).
For example, examine the following array:
>>> x = np.array([2, 1, 3, np.nan, 5, 2, 3, np.nan])
When we try to calculate the mean of the data, the result is undetermined:
>>> np.mean(x)
nan
The mean is calculated using roughly ``np.sum(x)/len(x)``, but since
any number added to ``NaN`` [1]_ produces ``NaN``, this doesn't work. Enter
masked arrays:
>>> m = np.ma.masked_array(x, np.isnan(x))
>>> m
masked_array(data=[2.0, 1.0, 3.0, --, 5.0, 2.0, 3.0, --],
mask=[False, False, False, True, False, False, False, True],
fill_value=1e+20)
Here, we construct a masked array that suppress all ``NaN`` values. We
may now proceed to calculate the mean of the other values:
>>> np.mean(m)
2.6666666666666665
.. [1] Not-a-Number, a floating point value that is the result of an
invalid operation.
.. moduleauthor:: Pierre Gerard-Marchant
.. moduleauthor:: Jarrod Millman
"""
from . import core, extras
from .core import *
from .extras import *
__all__ = ['core', 'extras']
__all__ += core.__all__
__all__ += extras.__all__
from numpy._pytesttester import PytestTester
test = PytestTester(__name__)
del PytestTester

View File

@ -0,0 +1,458 @@
from . import core, extras
from .core import (
MAError,
MaskedArray,
MaskError,
MaskType,
abs,
absolute,
add,
all,
allclose,
allequal,
alltrue,
amax,
amin,
angle,
anom,
anomalies,
any,
append,
arange,
arccos,
arccosh,
arcsin,
arcsinh,
arctan,
arctan2,
arctanh,
argmax,
argmin,
argsort,
around,
array,
asanyarray,
asarray,
bitwise_and,
bitwise_or,
bitwise_xor,
bool_,
ceil,
choose,
clip,
common_fill_value,
compress,
compressed,
concatenate,
conjugate,
convolve,
copy,
correlate,
cos,
cosh,
count,
cumprod,
cumsum,
default_fill_value,
diag,
diagonal,
diff,
divide,
empty,
empty_like,
equal,
exp,
expand_dims,
fabs,
filled,
fix_invalid,
flatten_mask,
flatten_structured_array,
floor,
floor_divide,
fmod,
frombuffer,
fromflex,
fromfunction,
getdata,
getmask,
getmaskarray,
greater,
greater_equal,
harden_mask,
hypot,
identity,
ids,
indices,
inner,
innerproduct,
is_mask,
is_masked,
isarray,
isMA,
isMaskedArray,
left_shift,
less,
less_equal,
log,
log2,
log10,
logical_and,
logical_not,
logical_or,
logical_xor,
make_mask,
make_mask_descr,
make_mask_none,
mask_or,
masked,
masked_array,
masked_equal,
masked_greater,
masked_greater_equal,
masked_inside,
masked_invalid,
masked_less,
masked_less_equal,
masked_not_equal,
masked_object,
masked_outside,
masked_print_option,
masked_singleton,
masked_values,
masked_where,
max,
maximum,
maximum_fill_value,
mean,
min,
minimum,
minimum_fill_value,
mod,
multiply,
mvoid,
ndim,
negative,
nomask,
nonzero,
not_equal,
ones,
ones_like,
outer,
outerproduct,
power,
prod,
product,
ptp,
put,
putmask,
ravel,
remainder,
repeat,
reshape,
resize,
right_shift,
round,
round_,
set_fill_value,
shape,
sin,
sinh,
size,
soften_mask,
sometrue,
sort,
sqrt,
squeeze,
std,
subtract,
sum,
swapaxes,
take,
tan,
tanh,
trace,
transpose,
true_divide,
var,
where,
zeros,
zeros_like,
)
from .extras import (
apply_along_axis,
apply_over_axes,
atleast_1d,
atleast_2d,
atleast_3d,
average,
clump_masked,
clump_unmasked,
column_stack,
compress_cols,
compress_nd,
compress_rowcols,
compress_rows,
corrcoef,
count_masked,
cov,
diagflat,
dot,
dstack,
ediff1d,
flatnotmasked_contiguous,
flatnotmasked_edges,
hsplit,
hstack,
in1d,
intersect1d,
isin,
mask_cols,
mask_rowcols,
mask_rows,
masked_all,
masked_all_like,
median,
mr_,
ndenumerate,
notmasked_contiguous,
notmasked_edges,
polyfit,
row_stack,
setdiff1d,
setxor1d,
stack,
union1d,
unique,
vander,
vstack,
)
__all__ = [
"core",
"extras",
"MAError",
"MaskError",
"MaskType",
"MaskedArray",
"abs",
"absolute",
"add",
"all",
"allclose",
"allequal",
"alltrue",
"amax",
"amin",
"angle",
"anom",
"anomalies",
"any",
"append",
"arange",
"arccos",
"arccosh",
"arcsin",
"arcsinh",
"arctan",
"arctan2",
"arctanh",
"argmax",
"argmin",
"argsort",
"around",
"array",
"asanyarray",
"asarray",
"bitwise_and",
"bitwise_or",
"bitwise_xor",
"bool_",
"ceil",
"choose",
"clip",
"common_fill_value",
"compress",
"compressed",
"concatenate",
"conjugate",
"convolve",
"copy",
"correlate",
"cos",
"cosh",
"count",
"cumprod",
"cumsum",
"default_fill_value",
"diag",
"diagonal",
"diff",
"divide",
"empty",
"empty_like",
"equal",
"exp",
"expand_dims",
"fabs",
"filled",
"fix_invalid",
"flatten_mask",
"flatten_structured_array",
"floor",
"floor_divide",
"fmod",
"frombuffer",
"fromflex",
"fromfunction",
"getdata",
"getmask",
"getmaskarray",
"greater",
"greater_equal",
"harden_mask",
"hypot",
"identity",
"ids",
"indices",
"inner",
"innerproduct",
"isMA",
"isMaskedArray",
"is_mask",
"is_masked",
"isarray",
"left_shift",
"less",
"less_equal",
"log",
"log10",
"log2",
"logical_and",
"logical_not",
"logical_or",
"logical_xor",
"make_mask",
"make_mask_descr",
"make_mask_none",
"mask_or",
"masked",
"masked_array",
"masked_equal",
"masked_greater",
"masked_greater_equal",
"masked_inside",
"masked_invalid",
"masked_less",
"masked_less_equal",
"masked_not_equal",
"masked_object",
"masked_outside",
"masked_print_option",
"masked_singleton",
"masked_values",
"masked_where",
"max",
"maximum",
"maximum_fill_value",
"mean",
"min",
"minimum",
"minimum_fill_value",
"mod",
"multiply",
"mvoid",
"ndim",
"negative",
"nomask",
"nonzero",
"not_equal",
"ones",
"ones_like",
"outer",
"outerproduct",
"power",
"prod",
"product",
"ptp",
"put",
"putmask",
"ravel",
"remainder",
"repeat",
"reshape",
"resize",
"right_shift",
"round",
"round_",
"set_fill_value",
"shape",
"sin",
"sinh",
"size",
"soften_mask",
"sometrue",
"sort",
"sqrt",
"squeeze",
"std",
"subtract",
"sum",
"swapaxes",
"take",
"tan",
"tanh",
"trace",
"transpose",
"true_divide",
"var",
"where",
"zeros",
"zeros_like",
"apply_along_axis",
"apply_over_axes",
"atleast_1d",
"atleast_2d",
"atleast_3d",
"average",
"clump_masked",
"clump_unmasked",
"column_stack",
"compress_cols",
"compress_nd",
"compress_rowcols",
"compress_rows",
"count_masked",
"corrcoef",
"cov",
"diagflat",
"dot",
"dstack",
"ediff1d",
"flatnotmasked_contiguous",
"flatnotmasked_edges",
"hsplit",
"hstack",
"isin",
"in1d",
"intersect1d",
"mask_cols",
"mask_rowcols",
"mask_rows",
"masked_all",
"masked_all_like",
"median",
"mr_",
"ndenumerate",
"notmasked_contiguous",
"notmasked_edges",
"polyfit",
"row_stack",
"setdiff1d",
"setxor1d",
"stack",
"unique",
"union1d",
"vander",
"vstack",
]

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,134 @@
from _typeshed import Incomplete
import numpy as np
from numpy.lib._function_base_impl import average
from numpy.lib._index_tricks_impl import AxisConcatenator
from .core import MaskedArray, dot
__all__ = [
"apply_along_axis",
"apply_over_axes",
"atleast_1d",
"atleast_2d",
"atleast_3d",
"average",
"clump_masked",
"clump_unmasked",
"column_stack",
"compress_cols",
"compress_nd",
"compress_rowcols",
"compress_rows",
"corrcoef",
"count_masked",
"cov",
"diagflat",
"dot",
"dstack",
"ediff1d",
"flatnotmasked_contiguous",
"flatnotmasked_edges",
"hsplit",
"hstack",
"in1d",
"intersect1d",
"isin",
"mask_cols",
"mask_rowcols",
"mask_rows",
"masked_all",
"masked_all_like",
"median",
"mr_",
"ndenumerate",
"notmasked_contiguous",
"notmasked_edges",
"polyfit",
"row_stack",
"setdiff1d",
"setxor1d",
"stack",
"union1d",
"unique",
"vander",
"vstack",
]
def count_masked(arr, axis=...): ...
def masked_all(shape, dtype=...): ...
def masked_all_like(arr): ...
class _fromnxfunction:
__name__: Incomplete
__doc__: Incomplete
def __init__(self, funcname) -> None: ...
def getdoc(self): ...
def __call__(self, *args, **params): ...
class _fromnxfunction_single(_fromnxfunction):
def __call__(self, x, *args, **params): ...
class _fromnxfunction_seq(_fromnxfunction):
def __call__(self, x, *args, **params): ...
class _fromnxfunction_allargs(_fromnxfunction):
def __call__(self, *args, **params): ...
atleast_1d: _fromnxfunction_allargs
atleast_2d: _fromnxfunction_allargs
atleast_3d: _fromnxfunction_allargs
vstack: _fromnxfunction_seq
row_stack: _fromnxfunction_seq
hstack: _fromnxfunction_seq
column_stack: _fromnxfunction_seq
dstack: _fromnxfunction_seq
stack: _fromnxfunction_seq
hsplit: _fromnxfunction_single
diagflat: _fromnxfunction_single
def apply_along_axis(func1d, axis, arr, *args, **kwargs): ...
def apply_over_axes(func, a, axes): ...
def median(a, axis=..., out=..., overwrite_input=..., keepdims=...): ...
def compress_nd(x, axis=...): ...
def compress_rowcols(x, axis=...): ...
def compress_rows(a): ...
def compress_cols(a): ...
def mask_rows(a, axis=...): ...
def mask_cols(a, axis=...): ...
def ediff1d(arr, to_end=..., to_begin=...): ...
def unique(ar1, return_index=..., return_inverse=...): ...
def intersect1d(ar1, ar2, assume_unique=...): ...
def setxor1d(ar1, ar2, assume_unique=...): ...
def in1d(ar1, ar2, assume_unique=..., invert=...): ...
def isin(element, test_elements, assume_unique=..., invert=...): ...
def union1d(ar1, ar2): ...
def setdiff1d(ar1, ar2, assume_unique=...): ...
def cov(x, y=..., rowvar=..., bias=..., allow_masked=..., ddof=...): ...
def corrcoef(x, y=..., rowvar=..., bias=..., allow_masked=..., ddof=...): ...
class MAxisConcatenator(AxisConcatenator):
@staticmethod
def concatenate(arrays: Incomplete, axis: int = 0) -> Incomplete: ... # type: ignore[override] # pyright: ignore[reportIncompatibleMethodOverride]
@classmethod
def makemat(cls, arr: Incomplete) -> Incomplete: ... # type: ignore[override] # pyright: ignore[reportIncompatibleVariableOverride]
class mr_class(MAxisConcatenator):
def __init__(self) -> None: ...
mr_: mr_class
def ndenumerate(a, compressed=...): ...
def flatnotmasked_edges(a): ...
def notmasked_edges(a, axis=...): ...
def flatnotmasked_contiguous(a): ...
def notmasked_contiguous(a, axis=...): ...
def clump_unmasked(a): ...
def clump_masked(a): ...
def vander(x, n=...): ...
def polyfit(x, y, deg, rcond=..., full=..., w=..., cov=...): ...
#
def mask_rowcols(a: Incomplete, axis: Incomplete | None = None) -> MaskedArray[Incomplete, np.dtype[Incomplete]]: ...

View File

@ -0,0 +1,773 @@
""":mod:`numpy.ma..mrecords`
Defines the equivalent of :class:`numpy.recarrays` for masked arrays,
where fields can be accessed as attributes.
Note that :class:`numpy.ma.MaskedArray` already supports structured datatypes
and the masking of individual fields.
.. moduleauthor:: Pierre Gerard-Marchant
"""
# We should make sure that no field is called '_mask','mask','_fieldmask',
# or whatever restricted keywords. An idea would be to no bother in the
# first place, and then rename the invalid fields with a trailing
# underscore. Maybe we could just overload the parser function ?
import warnings
import numpy as np
import numpy.ma as ma
_byteorderconv = np._core.records._byteorderconv
_check_fill_value = ma.core._check_fill_value
__all__ = [
'MaskedRecords', 'mrecarray', 'fromarrays', 'fromrecords',
'fromtextfile', 'addfield',
]
reserved_fields = ['_data', '_mask', '_fieldmask', 'dtype']
def _checknames(descr, names=None):
"""
Checks that field names ``descr`` are not reserved keywords.
If this is the case, a default 'f%i' is substituted. If the argument
`names` is not None, updates the field names to valid names.
"""
ndescr = len(descr)
default_names = [f'f{i}' for i in range(ndescr)]
if names is None:
new_names = default_names
else:
if isinstance(names, (tuple, list)):
new_names = names
elif isinstance(names, str):
new_names = names.split(',')
else:
raise NameError(f'illegal input names {names!r}')
nnames = len(new_names)
if nnames < ndescr:
new_names += default_names[nnames:]
ndescr = []
for (n, d, t) in zip(new_names, default_names, descr.descr):
if n in reserved_fields:
if t[0] in reserved_fields:
ndescr.append((d, t[1]))
else:
ndescr.append(t)
else:
ndescr.append((n, t[1]))
return np.dtype(ndescr)
def _get_fieldmask(self):
mdescr = [(n, '|b1') for n in self.dtype.names]
fdmask = np.empty(self.shape, dtype=mdescr)
fdmask.flat = tuple([False] * len(mdescr))
return fdmask
class MaskedRecords(ma.MaskedArray):
"""
Attributes
----------
_data : recarray
Underlying data, as a record array.
_mask : boolean array
Mask of the records. A record is masked when all its fields are
masked.
_fieldmask : boolean recarray
Record array of booleans, setting the mask of each individual field
of each record.
_fill_value : record
Filling values for each field.
"""
def __new__(cls, shape, dtype=None, buf=None, offset=0, strides=None,
formats=None, names=None, titles=None,
byteorder=None, aligned=False,
mask=ma.nomask, hard_mask=False, fill_value=None, keep_mask=True,
copy=False,
**options):
self = np.recarray.__new__(cls, shape, dtype=dtype, buf=buf, offset=offset,
strides=strides, formats=formats, names=names,
titles=titles, byteorder=byteorder,
aligned=aligned,)
mdtype = ma.make_mask_descr(self.dtype)
if mask is ma.nomask or not np.size(mask):
if not keep_mask:
self._mask = tuple([False] * len(mdtype))
else:
mask = np.array(mask, copy=copy)
if mask.shape != self.shape:
(nd, nm) = (self.size, mask.size)
if nm == 1:
mask = np.resize(mask, self.shape)
elif nm == nd:
mask = np.reshape(mask, self.shape)
else:
msg = (f"Mask and data not compatible: data size is {nd},"
" mask size is {nm}.")
raise ma.MAError(msg)
if not keep_mask:
self.__setmask__(mask)
self._sharedmask = True
else:
if mask.dtype == mdtype:
_mask = mask
else:
_mask = np.array([tuple([m] * len(mdtype)) for m in mask],
dtype=mdtype)
self._mask = _mask
return self
def __array_finalize__(self, obj):
# Make sure we have a _fieldmask by default
_mask = getattr(obj, '_mask', None)
if _mask is None:
objmask = getattr(obj, '_mask', ma.nomask)
_dtype = np.ndarray.__getattribute__(self, 'dtype')
if objmask is ma.nomask:
_mask = ma.make_mask_none(self.shape, dtype=_dtype)
else:
mdescr = ma.make_mask_descr(_dtype)
_mask = np.array([tuple([m] * len(mdescr)) for m in objmask],
dtype=mdescr).view(np.recarray)
# Update some of the attributes
_dict = self.__dict__
_dict.update(_mask=_mask)
self._update_from(obj)
if _dict['_baseclass'] == np.ndarray:
_dict['_baseclass'] = np.recarray
@property
def _data(self):
"""
Returns the data as a recarray.
"""
return np.ndarray.view(self, np.recarray)
@property
def _fieldmask(self):
"""
Alias to mask.
"""
return self._mask
def __len__(self):
"""
Returns the length
"""
# We have more than one record
if self.ndim:
return len(self._data)
# We have only one record: return the nb of fields
return len(self.dtype)
def __getattribute__(self, attr):
try:
return object.__getattribute__(self, attr)
except AttributeError:
# attr must be a fieldname
pass
fielddict = np.ndarray.__getattribute__(self, 'dtype').fields
try:
res = fielddict[attr][:2]
except (TypeError, KeyError) as e:
raise AttributeError(
f'record array has no attribute {attr}') from e
# So far, so good
_localdict = np.ndarray.__getattribute__(self, '__dict__')
_data = np.ndarray.view(self, _localdict['_baseclass'])
obj = _data.getfield(*res)
if obj.dtype.names is not None:
raise NotImplementedError("MaskedRecords is currently limited to"
"simple records.")
# Get some special attributes
# Reset the object's mask
hasmasked = False
_mask = _localdict.get('_mask', None)
if _mask is not None:
try:
_mask = _mask[attr]
except IndexError:
# Couldn't find a mask: use the default (nomask)
pass
tp_len = len(_mask.dtype)
hasmasked = _mask.view((bool, ((tp_len,) if tp_len else ()))).any()
if (obj.shape or hasmasked):
obj = obj.view(ma.MaskedArray)
obj._baseclass = np.ndarray
obj._isfield = True
obj._mask = _mask
# Reset the field values
_fill_value = _localdict.get('_fill_value', None)
if _fill_value is not None:
try:
obj._fill_value = _fill_value[attr]
except ValueError:
obj._fill_value = None
else:
obj = obj.item()
return obj
def __setattr__(self, attr, val):
"""
Sets the attribute attr to the value val.
"""
# Should we call __setmask__ first ?
if attr in ['mask', 'fieldmask']:
self.__setmask__(val)
return
# Create a shortcut (so that we don't have to call getattr all the time)
_localdict = object.__getattribute__(self, '__dict__')
# Check whether we're creating a new field
newattr = attr not in _localdict
try:
# Is attr a generic attribute ?
ret = object.__setattr__(self, attr, val)
except Exception:
# Not a generic attribute: exit if it's not a valid field
fielddict = np.ndarray.__getattribute__(self, 'dtype').fields or {}
optinfo = np.ndarray.__getattribute__(self, '_optinfo') or {}
if not (attr in fielddict or attr in optinfo):
raise
else:
# Get the list of names
fielddict = np.ndarray.__getattribute__(self, 'dtype').fields or {}
# Check the attribute
if attr not in fielddict:
return ret
if newattr:
# We just added this one or this setattr worked on an
# internal attribute.
try:
object.__delattr__(self, attr)
except Exception:
return ret
# Let's try to set the field
try:
res = fielddict[attr][:2]
except (TypeError, KeyError) as e:
raise AttributeError(
f'record array has no attribute {attr}') from e
if val is ma.masked:
_fill_value = _localdict['_fill_value']
if _fill_value is not None:
dval = _localdict['_fill_value'][attr]
else:
dval = val
mval = True
else:
dval = ma.filled(val)
mval = ma.getmaskarray(val)
obj = np.ndarray.__getattribute__(self, '_data').setfield(dval, *res)
_localdict['_mask'].__setitem__(attr, mval)
return obj
def __getitem__(self, indx):
"""
Returns all the fields sharing the same fieldname base.
The fieldname base is either `_data` or `_mask`.
"""
_localdict = self.__dict__
_mask = np.ndarray.__getattribute__(self, '_mask')
_data = np.ndarray.view(self, _localdict['_baseclass'])
# We want a field
if isinstance(indx, str):
# Make sure _sharedmask is True to propagate back to _fieldmask
# Don't use _set_mask, there are some copies being made that
# break propagation Don't force the mask to nomask, that wreaks
# easy masking
obj = _data[indx].view(ma.MaskedArray)
obj._mask = _mask[indx]
obj._sharedmask = True
fval = _localdict['_fill_value']
if fval is not None:
obj._fill_value = fval[indx]
# Force to masked if the mask is True
if not obj.ndim and obj._mask:
return ma.masked
return obj
# We want some elements.
# First, the data.
obj = np.asarray(_data[indx]).view(mrecarray)
obj._mask = np.asarray(_mask[indx]).view(np.recarray)
return obj
def __setitem__(self, indx, value):
"""
Sets the given record to value.
"""
ma.MaskedArray.__setitem__(self, indx, value)
if isinstance(indx, str):
self._mask[indx] = ma.getmaskarray(value)
def __str__(self):
"""
Calculates the string representation.
"""
if self.size > 1:
mstr = [f"({','.join([str(i) for i in s])})"
for s in zip(*[getattr(self, f) for f in self.dtype.names])]
return f"[{', '.join(mstr)}]"
else:
mstr = [f"{','.join([str(i) for i in s])}"
for s in zip([getattr(self, f) for f in self.dtype.names])]
return f"({', '.join(mstr)})"
def __repr__(self):
"""
Calculates the repr representation.
"""
_names = self.dtype.names
fmt = f"%{max(len(n) for n in _names) + 4}s : %s"
reprstr = [fmt % (f, getattr(self, f)) for f in self.dtype.names]
reprstr.insert(0, 'masked_records(')
reprstr.extend([fmt % (' fill_value', self.fill_value),
' )'])
return str("\n".join(reprstr))
def view(self, dtype=None, type=None):
"""
Returns a view of the mrecarray.
"""
# OK, basic copy-paste from MaskedArray.view.
if dtype is None:
if type is None:
output = np.ndarray.view(self)
else:
output = np.ndarray.view(self, type)
# Here again.
elif type is None:
try:
if issubclass(dtype, np.ndarray):
output = np.ndarray.view(self, dtype)
else:
output = np.ndarray.view(self, dtype)
# OK, there's the change
except TypeError:
dtype = np.dtype(dtype)
# we need to revert to MaskedArray, but keeping the possibility
# of subclasses (eg, TimeSeriesRecords), so we'll force a type
# set to the first parent
if dtype.fields is None:
basetype = self.__class__.__bases__[0]
output = self.__array__().view(dtype, basetype)
output._update_from(self)
else:
output = np.ndarray.view(self, dtype)
output._fill_value = None
else:
output = np.ndarray.view(self, dtype, type)
# Update the mask, just like in MaskedArray.view
if (getattr(output, '_mask', ma.nomask) is not ma.nomask):
mdtype = ma.make_mask_descr(output.dtype)
output._mask = self._mask.view(mdtype, np.ndarray)
output._mask.shape = output.shape
return output
def harden_mask(self):
"""
Forces the mask to hard.
"""
self._hardmask = True
def soften_mask(self):
"""
Forces the mask to soft
"""
self._hardmask = False
def copy(self):
"""
Returns a copy of the masked record.
"""
copied = self._data.copy().view(type(self))
copied._mask = self._mask.copy()
return copied
def tolist(self, fill_value=None):
"""
Return the data portion of the array as a list.
Data items are converted to the nearest compatible Python type.
Masked values are converted to fill_value. If fill_value is None,
the corresponding entries in the output list will be ``None``.
"""
if fill_value is not None:
return self.filled(fill_value).tolist()
result = np.array(self.filled().tolist(), dtype=object)
mask = np.array(self._mask.tolist())
result[mask] = None
return result.tolist()
def __getstate__(self):
"""Return the internal state of the masked array.
This is for pickling.
"""
state = (1,
self.shape,
self.dtype,
self.flags.fnc,
self._data.tobytes(),
self._mask.tobytes(),
self._fill_value,
)
return state
def __setstate__(self, state):
"""
Restore the internal state of the masked array.
This is for pickling. ``state`` is typically the output of the
``__getstate__`` output, and is a 5-tuple:
- class name
- a tuple giving the shape of the data
- a typecode for the data
- a binary string for the data
- a binary string for the mask.
"""
(ver, shp, typ, isf, raw, msk, flv) = state
np.ndarray.__setstate__(self, (shp, typ, isf, raw))
mdtype = np.dtype([(k, np.bool) for (k, _) in self.dtype.descr])
self.__dict__['_mask'].__setstate__((shp, mdtype, isf, msk))
self.fill_value = flv
def __reduce__(self):
"""
Return a 3-tuple for pickling a MaskedArray.
"""
return (_mrreconstruct,
(self.__class__, self._baseclass, (0,), 'b',),
self.__getstate__())
def _mrreconstruct(subtype, baseclass, baseshape, basetype,):
"""
Build a new MaskedArray from the information stored in a pickle.
"""
_data = np.ndarray.__new__(baseclass, baseshape, basetype).view(subtype)
_mask = np.ndarray.__new__(np.ndarray, baseshape, 'b1')
return subtype.__new__(subtype, _data, mask=_mask, dtype=basetype,)
mrecarray = MaskedRecords
###############################################################################
# Constructors #
###############################################################################
def fromarrays(arraylist, dtype=None, shape=None, formats=None,
names=None, titles=None, aligned=False, byteorder=None,
fill_value=None):
"""
Creates a mrecarray from a (flat) list of masked arrays.
Parameters
----------
arraylist : sequence
A list of (masked) arrays. Each element of the sequence is first converted
to a masked array if needed. If a 2D array is passed as argument, it is
processed line by line
dtype : {None, dtype}, optional
Data type descriptor.
shape : {None, integer}, optional
Number of records. If None, shape is defined from the shape of the
first array in the list.
formats : {None, sequence}, optional
Sequence of formats for each individual field. If None, the formats will
be autodetected by inspecting the fields and selecting the highest dtype
possible.
names : {None, sequence}, optional
Sequence of the names of each field.
fill_value : {None, sequence}, optional
Sequence of data to be used as filling values.
Notes
-----
Lists of tuples should be preferred over lists of lists for faster processing.
"""
datalist = [ma.getdata(x) for x in arraylist]
masklist = [np.atleast_1d(ma.getmaskarray(x)) for x in arraylist]
_array = np.rec.fromarrays(datalist,
dtype=dtype, shape=shape, formats=formats,
names=names, titles=titles, aligned=aligned,
byteorder=byteorder).view(mrecarray)
_array._mask.flat = list(zip(*masklist))
if fill_value is not None:
_array.fill_value = fill_value
return _array
def fromrecords(reclist, dtype=None, shape=None, formats=None, names=None,
titles=None, aligned=False, byteorder=None,
fill_value=None, mask=ma.nomask):
"""
Creates a MaskedRecords from a list of records.
Parameters
----------
reclist : sequence
A list of records. Each element of the sequence is first converted
to a masked array if needed. If a 2D array is passed as argument, it is
processed line by line
dtype : {None, dtype}, optional
Data type descriptor.
shape : {None,int}, optional
Number of records. If None, ``shape`` is defined from the shape of the
first array in the list.
formats : {None, sequence}, optional
Sequence of formats for each individual field. If None, the formats will
be autodetected by inspecting the fields and selecting the highest dtype
possible.
names : {None, sequence}, optional
Sequence of the names of each field.
fill_value : {None, sequence}, optional
Sequence of data to be used as filling values.
mask : {nomask, sequence}, optional.
External mask to apply on the data.
Notes
-----
Lists of tuples should be preferred over lists of lists for faster processing.
"""
# Grab the initial _fieldmask, if needed:
_mask = getattr(reclist, '_mask', None)
# Get the list of records.
if isinstance(reclist, np.ndarray):
# Make sure we don't have some hidden mask
if isinstance(reclist, ma.MaskedArray):
reclist = reclist.filled().view(np.ndarray)
# Grab the initial dtype, just in case
if dtype is None:
dtype = reclist.dtype
reclist = reclist.tolist()
mrec = np.rec.fromrecords(reclist, dtype=dtype, shape=shape, formats=formats,
names=names, titles=titles,
aligned=aligned, byteorder=byteorder).view(mrecarray)
# Set the fill_value if needed
if fill_value is not None:
mrec.fill_value = fill_value
# Now, let's deal w/ the mask
if mask is not ma.nomask:
mask = np.asarray(mask)
maskrecordlength = len(mask.dtype)
if maskrecordlength:
mrec._mask.flat = mask
elif mask.ndim == 2:
mrec._mask.flat = [tuple(m) for m in mask]
else:
mrec.__setmask__(mask)
if _mask is not None:
mrec._mask[:] = _mask
return mrec
def _guessvartypes(arr):
"""
Tries to guess the dtypes of the str_ ndarray `arr`.
Guesses by testing element-wise conversion. Returns a list of dtypes.
The array is first converted to ndarray. If the array is 2D, the test
is performed on the first line. An exception is raised if the file is
3D or more.
"""
vartypes = []
arr = np.asarray(arr)
if arr.ndim == 2:
arr = arr[0]
elif arr.ndim > 2:
raise ValueError("The array should be 2D at most!")
# Start the conversion loop.
for f in arr:
try:
int(f)
except (ValueError, TypeError):
try:
float(f)
except (ValueError, TypeError):
try:
complex(f)
except (ValueError, TypeError):
vartypes.append(arr.dtype)
else:
vartypes.append(np.dtype(complex))
else:
vartypes.append(np.dtype(float))
else:
vartypes.append(np.dtype(int))
return vartypes
def openfile(fname):
"""
Opens the file handle of file `fname`.
"""
# A file handle
if hasattr(fname, 'readline'):
return fname
# Try to open the file and guess its type
try:
f = open(fname)
except FileNotFoundError as e:
raise FileNotFoundError(f"No such file: '{fname}'") from e
if f.readline()[:2] != "\\x":
f.seek(0, 0)
return f
f.close()
raise NotImplementedError("Wow, binary file")
def fromtextfile(fname, delimiter=None, commentchar='#', missingchar='',
varnames=None, vartypes=None,
*, delimitor=np._NoValue): # backwards compatibility
"""
Creates a mrecarray from data stored in the file `filename`.
Parameters
----------
fname : {file name/handle}
Handle of an opened file.
delimiter : {None, string}, optional
Alphanumeric character used to separate columns in the file.
If None, any (group of) white spacestring(s) will be used.
commentchar : {'#', string}, optional
Alphanumeric character used to mark the start of a comment.
missingchar : {'', string}, optional
String indicating missing data, and used to create the masks.
varnames : {None, sequence}, optional
Sequence of the variable names. If None, a list will be created from
the first non empty line of the file.
vartypes : {None, sequence}, optional
Sequence of the variables dtypes. If None, it will be estimated from
the first non-commented line.
Ultra simple: the varnames are in the header, one line"""
if delimitor is not np._NoValue:
if delimiter is not None:
raise TypeError("fromtextfile() got multiple values for argument "
"'delimiter'")
# NumPy 1.22.0, 2021-09-23
warnings.warn("The 'delimitor' keyword argument of "
"numpy.ma.mrecords.fromtextfile() is deprecated "
"since NumPy 1.22.0, use 'delimiter' instead.",
DeprecationWarning, stacklevel=2)
delimiter = delimitor
# Try to open the file.
ftext = openfile(fname)
# Get the first non-empty line as the varnames
while True:
line = ftext.readline()
firstline = line[:line.find(commentchar)].strip()
_varnames = firstline.split(delimiter)
if len(_varnames) > 1:
break
if varnames is None:
varnames = _varnames
# Get the data.
_variables = ma.masked_array([line.strip().split(delimiter) for line in ftext
if line[0] != commentchar and len(line) > 1])
(_, nfields) = _variables.shape
ftext.close()
# Try to guess the dtype.
if vartypes is None:
vartypes = _guessvartypes(_variables[0])
else:
vartypes = [np.dtype(v) for v in vartypes]
if len(vartypes) != nfields:
msg = f"Attempting to {len(vartypes)} dtypes for {nfields} fields!"
msg += " Reverting to default."
warnings.warn(msg, stacklevel=2)
vartypes = _guessvartypes(_variables[0])
# Construct the descriptor.
mdescr = list(zip(varnames, vartypes))
mfillv = [ma.default_fill_value(f) for f in vartypes]
# Get the data and the mask.
# We just need a list of masked_arrays. It's easier to create it like that:
_mask = (_variables.T == missingchar)
_datalist = [ma.masked_array(a, mask=m, dtype=t, fill_value=f)
for (a, m, t, f) in zip(_variables.T, _mask, vartypes, mfillv)]
return fromarrays(_datalist, dtype=mdescr)
def addfield(mrecord, newfield, newfieldname=None):
"""Adds a new field to the masked record array
Uses `newfield` as data and `newfieldname` as name. If `newfieldname`
is None, the new field name is set to 'fi', where `i` is the number of
existing fields.
"""
_data = mrecord._data
_mask = mrecord._mask
if newfieldname is None or newfieldname in reserved_fields:
newfieldname = f'f{len(_data.dtype)}'
newfield = ma.array(newfield)
# Get the new data.
# Create a new empty recarray
newdtype = np.dtype(_data.dtype.descr + [(newfieldname, newfield.dtype)])
newdata = np.recarray(_data.shape, newdtype)
# Add the existing field
[newdata.setfield(_data.getfield(*f), *f)
for f in _data.dtype.fields.values()]
# Add the new field
newdata.setfield(newfield._data, *newdata.dtype.fields[newfieldname])
newdata = newdata.view(MaskedRecords)
# Get the new mask
# Create a new empty recarray
newmdtype = np.dtype([(n, np.bool) for n in newdtype.names])
newmask = np.recarray(_data.shape, newmdtype)
# Add the old masks
[newmask.setfield(_mask.getfield(*f), *f)
for f in _mask.dtype.fields.values()]
# Add the mask of the new field
newmask.setfield(ma.getmaskarray(newfield),
*newmask.dtype.fields[newfieldname])
newdata._mask = newmask
return newdata

View File

@ -0,0 +1,96 @@
from typing import Any, TypeVar
from numpy import dtype
from . import MaskedArray
__all__ = [
"MaskedRecords",
"mrecarray",
"fromarrays",
"fromrecords",
"fromtextfile",
"addfield",
]
_ShapeT_co = TypeVar("_ShapeT_co", covariant=True, bound=tuple[int, ...])
_DTypeT_co = TypeVar("_DTypeT_co", bound=dtype, covariant=True)
class MaskedRecords(MaskedArray[_ShapeT_co, _DTypeT_co]):
def __new__(
cls,
shape,
dtype=...,
buf=...,
offset=...,
strides=...,
formats=...,
names=...,
titles=...,
byteorder=...,
aligned=...,
mask=...,
hard_mask=...,
fill_value=...,
keep_mask=...,
copy=...,
**options,
): ...
_mask: Any
_fill_value: Any
@property
def _data(self): ...
@property
def _fieldmask(self): ...
def __array_finalize__(self, obj): ...
def __len__(self): ...
def __getattribute__(self, attr): ...
def __setattr__(self, attr, val): ...
def __getitem__(self, indx): ...
def __setitem__(self, indx, value): ...
def view(self, dtype=..., type=...): ...
def harden_mask(self): ...
def soften_mask(self): ...
def copy(self): ...
def tolist(self, fill_value=...): ...
def __reduce__(self): ...
mrecarray = MaskedRecords
def fromarrays(
arraylist,
dtype=...,
shape=...,
formats=...,
names=...,
titles=...,
aligned=...,
byteorder=...,
fill_value=...,
): ...
def fromrecords(
reclist,
dtype=...,
shape=...,
formats=...,
names=...,
titles=...,
aligned=...,
byteorder=...,
fill_value=...,
mask=...,
): ...
def fromtextfile(
fname,
delimiter=...,
commentchar=...,
missingchar=...,
varnames=...,
vartypes=...,
# NOTE: deprecated: NumPy 1.22.0, 2021-09-23
# delimitor=...,
): ...
def addfield(mrecord, newfield, newfieldname=...): ...

View File

@ -0,0 +1,40 @@
import pytest
import numpy as np
from numpy.ma import masked_array
from numpy.testing import assert_array_equal
def test_matrix_transpose_raises_error_for_1d():
msg = "matrix transpose with ndim < 2 is undefined"
ma_arr = masked_array(data=[1, 2, 3, 4, 5, 6],
mask=[1, 0, 1, 1, 1, 0])
with pytest.raises(ValueError, match=msg):
ma_arr.mT
def test_matrix_transpose_equals_transpose_2d():
ma_arr = masked_array(data=[[1, 2, 3], [4, 5, 6]],
mask=[[1, 0, 1], [1, 1, 0]])
assert_array_equal(ma_arr.T, ma_arr.mT)
ARRAY_SHAPES_TO_TEST = (
(5, 2),
(5, 2, 3),
(5, 2, 3, 4),
)
@pytest.mark.parametrize("shape", ARRAY_SHAPES_TO_TEST)
def test_matrix_transpose_equals_swapaxes(shape):
num_of_axes = len(shape)
vec = np.arange(shape[-1])
arr = np.broadcast_to(vec, shape)
rng = np.random.default_rng(42)
mask = rng.choice([0, 1], size=shape)
ma_arr = masked_array(data=arr, mask=mask)
tgt = np.swapaxes(arr, num_of_axes - 2, num_of_axes - 1)
assert_array_equal(tgt, ma_arr.mT)

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,87 @@
"""Test deprecation and future warnings.
"""
import io
import textwrap
import pytest
import numpy as np
from numpy.ma.core import MaskedArrayFutureWarning
from numpy.ma.testutils import assert_equal
from numpy.testing import assert_warns
class TestArgsort:
""" gh-8701 """
def _test_base(self, argsort, cls):
arr_0d = np.array(1).view(cls)
argsort(arr_0d)
arr_1d = np.array([1, 2, 3]).view(cls)
argsort(arr_1d)
# argsort has a bad default for >1d arrays
arr_2d = np.array([[1, 2], [3, 4]]).view(cls)
result = assert_warns(
np.ma.core.MaskedArrayFutureWarning, argsort, arr_2d)
assert_equal(result, argsort(arr_2d, axis=None))
# should be no warnings for explicitly specifying it
argsort(arr_2d, axis=None)
argsort(arr_2d, axis=-1)
def test_function_ndarray(self):
return self._test_base(np.ma.argsort, np.ndarray)
def test_function_maskedarray(self):
return self._test_base(np.ma.argsort, np.ma.MaskedArray)
def test_method(self):
return self._test_base(np.ma.MaskedArray.argsort, np.ma.MaskedArray)
class TestMinimumMaximum:
def test_axis_default(self):
# NumPy 1.13, 2017-05-06
data1d = np.ma.arange(6)
data2d = data1d.reshape(2, 3)
ma_min = np.ma.minimum.reduce
ma_max = np.ma.maximum.reduce
# check that the default axis is still None, but warns on 2d arrays
result = assert_warns(MaskedArrayFutureWarning, ma_max, data2d)
assert_equal(result, ma_max(data2d, axis=None))
result = assert_warns(MaskedArrayFutureWarning, ma_min, data2d)
assert_equal(result, ma_min(data2d, axis=None))
# no warnings on 1d, as both new and old defaults are equivalent
result = ma_min(data1d)
assert_equal(result, ma_min(data1d, axis=None))
assert_equal(result, ma_min(data1d, axis=0))
result = ma_max(data1d)
assert_equal(result, ma_max(data1d, axis=None))
assert_equal(result, ma_max(data1d, axis=0))
class TestFromtextfile:
def test_fromtextfile_delimitor(self):
# NumPy 1.22.0, 2021-09-23
textfile = io.StringIO(textwrap.dedent(
"""
A,B,C,D
'string 1';1;1.0;'mixed column'
'string 2';2;2.0;
'string 3';3;3.0;123
'string 4';4;4.0;3.14
"""
))
with pytest.warns(DeprecationWarning):
result = np.ma.mrecords.fromtextfile(textfile, delimitor=';')

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,497 @@
"""Tests suite for mrecords.
:author: Pierre Gerard-Marchant
:contact: pierregm_at_uga_dot_edu
"""
import pickle
import numpy as np
import numpy.ma as ma
from numpy._core.records import fromarrays as recfromarrays
from numpy._core.records import fromrecords as recfromrecords
from numpy._core.records import recarray
from numpy.ma import masked, nomask
from numpy.ma.mrecords import (
MaskedRecords,
addfield,
fromarrays,
fromrecords,
fromtextfile,
mrecarray,
)
from numpy.ma.testutils import (
assert_,
assert_equal,
assert_equal_records,
)
from numpy.testing import temppath
class TestMRecords:
ilist = [1, 2, 3, 4, 5]
flist = [1.1, 2.2, 3.3, 4.4, 5.5]
slist = [b'one', b'two', b'three', b'four', b'five']
ddtype = [('a', int), ('b', float), ('c', '|S8')]
mask = [0, 1, 0, 0, 1]
base = ma.array(list(zip(ilist, flist, slist)), mask=mask, dtype=ddtype)
def test_byview(self):
# Test creation by view
base = self.base
mbase = base.view(mrecarray)
assert_equal(mbase.recordmask, base.recordmask)
assert_equal_records(mbase._mask, base._mask)
assert_(isinstance(mbase._data, recarray))
assert_equal_records(mbase._data, base._data.view(recarray))
for field in ('a', 'b', 'c'):
assert_equal(base[field], mbase[field])
assert_equal_records(mbase.view(mrecarray), mbase)
def test_get(self):
# Tests fields retrieval
base = self.base.copy()
mbase = base.view(mrecarray)
# As fields..........
for field in ('a', 'b', 'c'):
assert_equal(getattr(mbase, field), mbase[field])
assert_equal(base[field], mbase[field])
# as elements .......
mbase_first = mbase[0]
assert_(isinstance(mbase_first, mrecarray))
assert_equal(mbase_first.dtype, mbase.dtype)
assert_equal(mbase_first.tolist(), (1, 1.1, b'one'))
# Used to be mask, now it's recordmask
assert_equal(mbase_first.recordmask, nomask)
assert_equal(mbase_first._mask.item(), (False, False, False))
assert_equal(mbase_first['a'], mbase['a'][0])
mbase_last = mbase[-1]
assert_(isinstance(mbase_last, mrecarray))
assert_equal(mbase_last.dtype, mbase.dtype)
assert_equal(mbase_last.tolist(), (None, None, None))
# Used to be mask, now it's recordmask
assert_equal(mbase_last.recordmask, True)
assert_equal(mbase_last._mask.item(), (True, True, True))
assert_equal(mbase_last['a'], mbase['a'][-1])
assert_(mbase_last['a'] is masked)
# as slice ..........
mbase_sl = mbase[:2]
assert_(isinstance(mbase_sl, mrecarray))
assert_equal(mbase_sl.dtype, mbase.dtype)
# Used to be mask, now it's recordmask
assert_equal(mbase_sl.recordmask, [0, 1])
assert_equal_records(mbase_sl.mask,
np.array([(False, False, False),
(True, True, True)],
dtype=mbase._mask.dtype))
assert_equal_records(mbase_sl, base[:2].view(mrecarray))
for field in ('a', 'b', 'c'):
assert_equal(getattr(mbase_sl, field), base[:2][field])
def test_set_fields(self):
# Tests setting fields.
base = self.base.copy()
mbase = base.view(mrecarray)
mbase = mbase.copy()
mbase.fill_value = (999999, 1e20, 'N/A')
# Change the data, the mask should be conserved
mbase.a._data[:] = 5
assert_equal(mbase['a']._data, [5, 5, 5, 5, 5])
assert_equal(mbase['a']._mask, [0, 1, 0, 0, 1])
# Change the elements, and the mask will follow
mbase.a = 1
assert_equal(mbase['a']._data, [1] * 5)
assert_equal(ma.getmaskarray(mbase['a']), [0] * 5)
# Use to be _mask, now it's recordmask
assert_equal(mbase.recordmask, [False] * 5)
assert_equal(mbase._mask.tolist(),
np.array([(0, 0, 0),
(0, 1, 1),
(0, 0, 0),
(0, 0, 0),
(0, 1, 1)],
dtype=bool))
# Set a field to mask ........................
mbase.c = masked
# Use to be mask, and now it's still mask !
assert_equal(mbase.c.mask, [1] * 5)
assert_equal(mbase.c.recordmask, [1] * 5)
assert_equal(ma.getmaskarray(mbase['c']), [1] * 5)
assert_equal(ma.getdata(mbase['c']), [b'N/A'] * 5)
assert_equal(mbase._mask.tolist(),
np.array([(0, 0, 1),
(0, 1, 1),
(0, 0, 1),
(0, 0, 1),
(0, 1, 1)],
dtype=bool))
# Set fields by slices .......................
mbase = base.view(mrecarray).copy()
mbase.a[3:] = 5
assert_equal(mbase.a, [1, 2, 3, 5, 5])
assert_equal(mbase.a._mask, [0, 1, 0, 0, 0])
mbase.b[3:] = masked
assert_equal(mbase.b, base['b'])
assert_equal(mbase.b._mask, [0, 1, 0, 1, 1])
# Set fields globally..........................
ndtype = [('alpha', '|S1'), ('num', int)]
data = ma.array([('a', 1), ('b', 2), ('c', 3)], dtype=ndtype)
rdata = data.view(MaskedRecords)
val = ma.array([10, 20, 30], mask=[1, 0, 0])
rdata['num'] = val
assert_equal(rdata.num, val)
assert_equal(rdata.num.mask, [1, 0, 0])
def test_set_fields_mask(self):
# Tests setting the mask of a field.
base = self.base.copy()
# This one has already a mask....
mbase = base.view(mrecarray)
mbase['a'][-2] = masked
assert_equal(mbase.a, [1, 2, 3, 4, 5])
assert_equal(mbase.a._mask, [0, 1, 0, 1, 1])
# This one has not yet
mbase = fromarrays([np.arange(5), np.random.rand(5)],
dtype=[('a', int), ('b', float)])
mbase['a'][-2] = masked
assert_equal(mbase.a, [0, 1, 2, 3, 4])
assert_equal(mbase.a._mask, [0, 0, 0, 1, 0])
def test_set_mask(self):
base = self.base.copy()
mbase = base.view(mrecarray)
# Set the mask to True .......................
mbase.mask = masked
assert_equal(ma.getmaskarray(mbase['b']), [1] * 5)
assert_equal(mbase['a']._mask, mbase['b']._mask)
assert_equal(mbase['a']._mask, mbase['c']._mask)
assert_equal(mbase._mask.tolist(),
np.array([(1, 1, 1)] * 5, dtype=bool))
# Delete the mask ............................
mbase.mask = nomask
assert_equal(ma.getmaskarray(mbase['c']), [0] * 5)
assert_equal(mbase._mask.tolist(),
np.array([(0, 0, 0)] * 5, dtype=bool))
def test_set_mask_fromarray(self):
base = self.base.copy()
mbase = base.view(mrecarray)
# Sets the mask w/ an array
mbase.mask = [1, 0, 0, 0, 1]
assert_equal(mbase.a.mask, [1, 0, 0, 0, 1])
assert_equal(mbase.b.mask, [1, 0, 0, 0, 1])
assert_equal(mbase.c.mask, [1, 0, 0, 0, 1])
# Yay, once more !
mbase.mask = [0, 0, 0, 0, 1]
assert_equal(mbase.a.mask, [0, 0, 0, 0, 1])
assert_equal(mbase.b.mask, [0, 0, 0, 0, 1])
assert_equal(mbase.c.mask, [0, 0, 0, 0, 1])
def test_set_mask_fromfields(self):
mbase = self.base.copy().view(mrecarray)
nmask = np.array(
[(0, 1, 0), (0, 1, 0), (1, 0, 1), (1, 0, 1), (0, 0, 0)],
dtype=[('a', bool), ('b', bool), ('c', bool)])
mbase.mask = nmask
assert_equal(mbase.a.mask, [0, 0, 1, 1, 0])
assert_equal(mbase.b.mask, [1, 1, 0, 0, 0])
assert_equal(mbase.c.mask, [0, 0, 1, 1, 0])
# Reinitialize and redo
mbase.mask = False
mbase.fieldmask = nmask
assert_equal(mbase.a.mask, [0, 0, 1, 1, 0])
assert_equal(mbase.b.mask, [1, 1, 0, 0, 0])
assert_equal(mbase.c.mask, [0, 0, 1, 1, 0])
def test_set_elements(self):
base = self.base.copy()
# Set an element to mask .....................
mbase = base.view(mrecarray).copy()
mbase[-2] = masked
assert_equal(
mbase._mask.tolist(),
np.array([(0, 0, 0), (1, 1, 1), (0, 0, 0), (1, 1, 1), (1, 1, 1)],
dtype=bool))
# Used to be mask, now it's recordmask!
assert_equal(mbase.recordmask, [0, 1, 0, 1, 1])
# Set slices .................................
mbase = base.view(mrecarray).copy()
mbase[:2] = (5, 5, 5)
assert_equal(mbase.a._data, [5, 5, 3, 4, 5])
assert_equal(mbase.a._mask, [0, 0, 0, 0, 1])
assert_equal(mbase.b._data, [5., 5., 3.3, 4.4, 5.5])
assert_equal(mbase.b._mask, [0, 0, 0, 0, 1])
assert_equal(mbase.c._data,
[b'5', b'5', b'three', b'four', b'five'])
assert_equal(mbase.b._mask, [0, 0, 0, 0, 1])
mbase = base.view(mrecarray).copy()
mbase[:2] = masked
assert_equal(mbase.a._data, [1, 2, 3, 4, 5])
assert_equal(mbase.a._mask, [1, 1, 0, 0, 1])
assert_equal(mbase.b._data, [1.1, 2.2, 3.3, 4.4, 5.5])
assert_equal(mbase.b._mask, [1, 1, 0, 0, 1])
assert_equal(mbase.c._data,
[b'one', b'two', b'three', b'four', b'five'])
assert_equal(mbase.b._mask, [1, 1, 0, 0, 1])
def test_setslices_hardmask(self):
# Tests setting slices w/ hardmask.
base = self.base.copy()
mbase = base.view(mrecarray)
mbase.harden_mask()
try:
mbase[-2:] = (5, 5, 5)
assert_equal(mbase.a._data, [1, 2, 3, 5, 5])
assert_equal(mbase.b._data, [1.1, 2.2, 3.3, 5, 5.5])
assert_equal(mbase.c._data,
[b'one', b'two', b'three', b'5', b'five'])
assert_equal(mbase.a._mask, [0, 1, 0, 0, 1])
assert_equal(mbase.b._mask, mbase.a._mask)
assert_equal(mbase.b._mask, mbase.c._mask)
except NotImplementedError:
# OK, not implemented yet...
pass
except AssertionError:
raise
else:
raise Exception("Flexible hard masks should be supported !")
# Not using a tuple should crash
try:
mbase[-2:] = 3
except (NotImplementedError, TypeError):
pass
else:
raise TypeError("Should have expected a readable buffer object!")
def test_hardmask(self):
# Test hardmask
base = self.base.copy()
mbase = base.view(mrecarray)
mbase.harden_mask()
assert_(mbase._hardmask)
mbase.mask = nomask
assert_equal_records(mbase._mask, base._mask)
mbase.soften_mask()
assert_(not mbase._hardmask)
mbase.mask = nomask
# So, the mask of a field is no longer set to nomask...
assert_equal_records(mbase._mask,
ma.make_mask_none(base.shape, base.dtype))
assert_(ma.make_mask(mbase['b']._mask) is nomask)
assert_equal(mbase['a']._mask, mbase['b']._mask)
def test_pickling(self):
# Test pickling
base = self.base.copy()
mrec = base.view(mrecarray)
for proto in range(2, pickle.HIGHEST_PROTOCOL + 1):
_ = pickle.dumps(mrec, protocol=proto)
mrec_ = pickle.loads(_)
assert_equal(mrec_.dtype, mrec.dtype)
assert_equal_records(mrec_._data, mrec._data)
assert_equal(mrec_._mask, mrec._mask)
assert_equal_records(mrec_._mask, mrec._mask)
def test_filled(self):
# Test filling the array
_a = ma.array([1, 2, 3], mask=[0, 0, 1], dtype=int)
_b = ma.array([1.1, 2.2, 3.3], mask=[0, 0, 1], dtype=float)
_c = ma.array(['one', 'two', 'three'], mask=[0, 0, 1], dtype='|S8')
ddtype = [('a', int), ('b', float), ('c', '|S8')]
mrec = fromarrays([_a, _b, _c], dtype=ddtype,
fill_value=(99999, 99999., 'N/A'))
mrecfilled = mrec.filled()
assert_equal(mrecfilled['a'], np.array((1, 2, 99999), dtype=int))
assert_equal(mrecfilled['b'], np.array((1.1, 2.2, 99999.),
dtype=float))
assert_equal(mrecfilled['c'], np.array(('one', 'two', 'N/A'),
dtype='|S8'))
def test_tolist(self):
# Test tolist.
_a = ma.array([1, 2, 3], mask=[0, 0, 1], dtype=int)
_b = ma.array([1.1, 2.2, 3.3], mask=[0, 0, 1], dtype=float)
_c = ma.array(['one', 'two', 'three'], mask=[1, 0, 0], dtype='|S8')
ddtype = [('a', int), ('b', float), ('c', '|S8')]
mrec = fromarrays([_a, _b, _c], dtype=ddtype,
fill_value=(99999, 99999., 'N/A'))
assert_equal(mrec.tolist(),
[(1, 1.1, None), (2, 2.2, b'two'),
(None, None, b'three')])
def test_withnames(self):
# Test the creation w/ format and names
x = mrecarray(1, formats=float, names='base')
x[0]['base'] = 10
assert_equal(x['base'][0], 10)
def test_exotic_formats(self):
# Test that 'exotic' formats are processed properly
easy = mrecarray(1, dtype=[('i', int), ('s', '|S8'), ('f', float)])
easy[0] = masked
assert_equal(easy.filled(1).item(), (1, b'1', 1.))
solo = mrecarray(1, dtype=[('f0', '<f8', (2, 2))])
solo[0] = masked
assert_equal(solo.filled(1).item(),
np.array((1,), dtype=solo.dtype).item())
mult = mrecarray(2, dtype="i4, (2,3)float, float")
mult[0] = masked
mult[1] = (1, 1, 1)
mult.filled(0)
assert_equal_records(mult.filled(0),
np.array([(0, 0, 0), (1, 1, 1)],
dtype=mult.dtype))
class TestView:
def setup_method(self):
(a, b) = (np.arange(10), np.random.rand(10))
ndtype = [('a', float), ('b', float)]
arr = np.array(list(zip(a, b)), dtype=ndtype)
mrec = fromarrays([a, b], dtype=ndtype, fill_value=(-9., -99.))
mrec.mask[3] = (False, True)
self.data = (mrec, a, b, arr)
def test_view_by_itself(self):
(mrec, a, b, arr) = self.data
test = mrec.view()
assert_(isinstance(test, MaskedRecords))
assert_equal_records(test, mrec)
assert_equal_records(test._mask, mrec._mask)
def test_view_simple_dtype(self):
(mrec, a, b, arr) = self.data
ntype = (float, 2)
test = mrec.view(ntype)
assert_(isinstance(test, ma.MaskedArray))
assert_equal(test, np.array(list(zip(a, b)), dtype=float))
assert_(test[3, 1] is ma.masked)
def test_view_flexible_type(self):
(mrec, a, b, arr) = self.data
alttype = [('A', float), ('B', float)]
test = mrec.view(alttype)
assert_(isinstance(test, MaskedRecords))
assert_equal_records(test, arr.view(alttype))
assert_(test['B'][3] is masked)
assert_equal(test.dtype, np.dtype(alttype))
assert_(test._fill_value is None)
##############################################################################
class TestMRecordsImport:
_a = ma.array([1, 2, 3], mask=[0, 0, 1], dtype=int)
_b = ma.array([1.1, 2.2, 3.3], mask=[0, 0, 1], dtype=float)
_c = ma.array([b'one', b'two', b'three'],
mask=[0, 0, 1], dtype='|S8')
ddtype = [('a', int), ('b', float), ('c', '|S8')]
mrec = fromarrays([_a, _b, _c], dtype=ddtype,
fill_value=(b'99999', b'99999.',
b'N/A'))
nrec = recfromarrays((_a._data, _b._data, _c._data), dtype=ddtype)
data = (mrec, nrec, ddtype)
def test_fromarrays(self):
_a = ma.array([1, 2, 3], mask=[0, 0, 1], dtype=int)
_b = ma.array([1.1, 2.2, 3.3], mask=[0, 0, 1], dtype=float)
_c = ma.array(['one', 'two', 'three'], mask=[0, 0, 1], dtype='|S8')
(mrec, nrec, _) = self.data
for (f, l) in zip(('a', 'b', 'c'), (_a, _b, _c)):
assert_equal(getattr(mrec, f)._mask, l._mask)
# One record only
_x = ma.array([1, 1.1, 'one'], mask=[1, 0, 0], dtype=object)
assert_equal_records(fromarrays(_x, dtype=mrec.dtype), mrec[0])
def test_fromrecords(self):
# Test construction from records.
(mrec, nrec, ddtype) = self.data
# ......
palist = [(1, 'abc', 3.7000002861022949, 0),
(2, 'xy', 6.6999998092651367, 1),
(0, ' ', 0.40000000596046448, 0)]
pa = recfromrecords(palist, names='c1, c2, c3, c4')
mpa = fromrecords(palist, names='c1, c2, c3, c4')
assert_equal_records(pa, mpa)
# .....
_mrec = fromrecords(nrec)
assert_equal(_mrec.dtype, mrec.dtype)
for field in _mrec.dtype.names:
assert_equal(getattr(_mrec, field), getattr(mrec._data, field))
_mrec = fromrecords(nrec.tolist(), names='c1,c2,c3')
assert_equal(_mrec.dtype, [('c1', int), ('c2', float), ('c3', '|S5')])
for (f, n) in zip(('c1', 'c2', 'c3'), ('a', 'b', 'c')):
assert_equal(getattr(_mrec, f), getattr(mrec._data, n))
_mrec = fromrecords(mrec)
assert_equal(_mrec.dtype, mrec.dtype)
assert_equal_records(_mrec._data, mrec.filled())
assert_equal_records(_mrec._mask, mrec._mask)
def test_fromrecords_wmask(self):
# Tests construction from records w/ mask.
(mrec, nrec, ddtype) = self.data
_mrec = fromrecords(nrec.tolist(), dtype=ddtype, mask=[0, 1, 0,])
assert_equal_records(_mrec._data, mrec._data)
assert_equal(_mrec._mask.tolist(), [(0, 0, 0), (1, 1, 1), (0, 0, 0)])
_mrec = fromrecords(nrec.tolist(), dtype=ddtype, mask=True)
assert_equal_records(_mrec._data, mrec._data)
assert_equal(_mrec._mask.tolist(), [(1, 1, 1), (1, 1, 1), (1, 1, 1)])
_mrec = fromrecords(nrec.tolist(), dtype=ddtype, mask=mrec._mask)
assert_equal_records(_mrec._data, mrec._data)
assert_equal(_mrec._mask.tolist(), mrec._mask.tolist())
_mrec = fromrecords(nrec.tolist(), dtype=ddtype,
mask=mrec._mask.tolist())
assert_equal_records(_mrec._data, mrec._data)
assert_equal(_mrec._mask.tolist(), mrec._mask.tolist())
def test_fromtextfile(self):
# Tests reading from a text file.
fcontent = (
"""#
'One (S)','Two (I)','Three (F)','Four (M)','Five (-)','Six (C)'
'strings',1,1.0,'mixed column',,1
'with embedded "double quotes"',2,2.0,1.0,,1
'strings',3,3.0E5,3,,1
'strings',4,-1e-10,,,1
""")
with temppath() as path:
with open(path, 'w') as f:
f.write(fcontent)
mrectxt = fromtextfile(path, delimiter=',', varnames='ABCDEFG')
assert_(isinstance(mrectxt, MaskedRecords))
assert_equal(mrectxt.F, [1, 1, 1, 1])
assert_equal(mrectxt.E._mask, [1, 1, 1, 1])
assert_equal(mrectxt.C, [1, 2, 3.e+5, -1e-10])
def test_addfield(self):
# Tests addfield
(mrec, nrec, ddtype) = self.data
(d, m) = ([100, 200, 300], [1, 0, 0])
mrec = addfield(mrec, ma.array(d, mask=m))
assert_equal(mrec.f3, d)
assert_equal(mrec.f3._mask, m)
def test_record_array_with_object_field():
# Trac #1839
y = ma.masked_array(
[(1, '2'), (3, '4')],
mask=[(0, 0), (0, 1)],
dtype=[('a', int), ('b', object)])
# getting an item used to fail
y[1]

View File

@ -0,0 +1,942 @@
import pickle
from functools import reduce
import pytest
import numpy as np
import numpy._core.fromnumeric as fromnumeric
import numpy._core.umath as umath
from numpy.ma import (
MaskedArray,
MaskType,
absolute,
add,
all,
allclose,
allequal,
alltrue,
arange,
arccos,
arcsin,
arctan,
arctan2,
array,
average,
choose,
concatenate,
conjugate,
cos,
cosh,
count,
divide,
equal,
exp,
filled,
getmask,
greater,
greater_equal,
inner,
isMaskedArray,
less,
less_equal,
log,
log10,
make_mask,
masked,
masked_array,
masked_equal,
masked_greater,
masked_greater_equal,
masked_inside,
masked_less,
masked_less_equal,
masked_not_equal,
masked_outside,
masked_print_option,
masked_values,
masked_where,
maximum,
minimum,
multiply,
nomask,
nonzero,
not_equal,
ones,
outer,
product,
put,
ravel,
repeat,
resize,
shape,
sin,
sinh,
sometrue,
sort,
sqrt,
subtract,
sum,
take,
tan,
tanh,
transpose,
where,
zeros,
)
from numpy.testing import (
assert_,
assert_equal,
assert_raises,
)
pi = np.pi
def eq(v, w, msg=''):
result = allclose(v, w)
if not result:
print(f'Not eq:{msg}\n{v}\n----{w}')
return result
class TestMa:
def setup_method(self):
x = np.array([1., 1., 1., -2., pi / 2.0, 4., 5., -10., 10., 1., 2., 3.])
y = np.array([5., 0., 3., 2., -1., -4., 0., -10., 10., 1., 0., 3.])
a10 = 10.
m1 = [1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0]
m2 = [0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1]
xm = array(x, mask=m1)
ym = array(y, mask=m2)
z = np.array([-.5, 0., .5, .8])
zm = array(z, mask=[0, 1, 0, 0])
xf = np.where(m1, 1e+20, x)
s = x.shape
xm.set_fill_value(1e+20)
self.d = (x, y, a10, m1, m2, xm, ym, z, zm, xf, s)
def test_testBasic1d(self):
# Test of basic array creation and properties in 1 dimension.
(x, y, a10, m1, m2, xm, ym, z, zm, xf, s) = self.d
assert_(not isMaskedArray(x))
assert_(isMaskedArray(xm))
assert_equal(shape(xm), s)
assert_equal(xm.shape, s)
assert_equal(xm.dtype, x.dtype)
assert_equal(xm.size, reduce(lambda x, y: x * y, s))
assert_equal(count(xm), len(m1) - reduce(lambda x, y: x + y, m1))
assert_(eq(xm, xf))
assert_(eq(filled(xm, 1.e20), xf))
assert_(eq(x, xm))
@pytest.mark.parametrize("s", [(4, 3), (6, 2)])
def test_testBasic2d(self, s):
# Test of basic array creation and properties in 2 dimensions.
(x, y, a10, m1, m2, xm, ym, z, zm, xf, s) = self.d
x.shape = s
y.shape = s
xm.shape = s
ym.shape = s
xf.shape = s
assert_(not isMaskedArray(x))
assert_(isMaskedArray(xm))
assert_equal(shape(xm), s)
assert_equal(xm.shape, s)
assert_equal(xm.size, reduce(lambda x, y: x * y, s))
assert_equal(count(xm), len(m1) - reduce(lambda x, y: x + y, m1))
assert_(eq(xm, xf))
assert_(eq(filled(xm, 1.e20), xf))
assert_(eq(x, xm))
def test_testArithmetic(self):
# Test of basic arithmetic.
(x, y, a10, m1, m2, xm, ym, z, zm, xf, s) = self.d
a2d = array([[1, 2], [0, 4]])
a2dm = masked_array(a2d, [[0, 0], [1, 0]])
assert_(eq(a2d * a2d, a2d * a2dm))
assert_(eq(a2d + a2d, a2d + a2dm))
assert_(eq(a2d - a2d, a2d - a2dm))
for s in [(12,), (4, 3), (2, 6)]:
x = x.reshape(s)
y = y.reshape(s)
xm = xm.reshape(s)
ym = ym.reshape(s)
xf = xf.reshape(s)
assert_(eq(-x, -xm))
assert_(eq(x + y, xm + ym))
assert_(eq(x - y, xm - ym))
assert_(eq(x * y, xm * ym))
with np.errstate(divide='ignore', invalid='ignore'):
assert_(eq(x / y, xm / ym))
assert_(eq(a10 + y, a10 + ym))
assert_(eq(a10 - y, a10 - ym))
assert_(eq(a10 * y, a10 * ym))
with np.errstate(divide='ignore', invalid='ignore'):
assert_(eq(a10 / y, a10 / ym))
assert_(eq(x + a10, xm + a10))
assert_(eq(x - a10, xm - a10))
assert_(eq(x * a10, xm * a10))
assert_(eq(x / a10, xm / a10))
assert_(eq(x ** 2, xm ** 2))
assert_(eq(abs(x) ** 2.5, abs(xm) ** 2.5))
assert_(eq(x ** y, xm ** ym))
assert_(eq(np.add(x, y), add(xm, ym)))
assert_(eq(np.subtract(x, y), subtract(xm, ym)))
assert_(eq(np.multiply(x, y), multiply(xm, ym)))
with np.errstate(divide='ignore', invalid='ignore'):
assert_(eq(np.divide(x, y), divide(xm, ym)))
def test_testMixedArithmetic(self):
na = np.array([1])
ma = array([1])
assert_(isinstance(na + ma, MaskedArray))
assert_(isinstance(ma + na, MaskedArray))
def test_testUfuncs1(self):
# Test various functions such as sin, cos.
(x, y, a10, m1, m2, xm, ym, z, zm, xf, s) = self.d
assert_(eq(np.cos(x), cos(xm)))
assert_(eq(np.cosh(x), cosh(xm)))
assert_(eq(np.sin(x), sin(xm)))
assert_(eq(np.sinh(x), sinh(xm)))
assert_(eq(np.tan(x), tan(xm)))
assert_(eq(np.tanh(x), tanh(xm)))
with np.errstate(divide='ignore', invalid='ignore'):
assert_(eq(np.sqrt(abs(x)), sqrt(xm)))
assert_(eq(np.log(abs(x)), log(xm)))
assert_(eq(np.log10(abs(x)), log10(xm)))
assert_(eq(np.exp(x), exp(xm)))
assert_(eq(np.arcsin(z), arcsin(zm)))
assert_(eq(np.arccos(z), arccos(zm)))
assert_(eq(np.arctan(z), arctan(zm)))
assert_(eq(np.arctan2(x, y), arctan2(xm, ym)))
assert_(eq(np.absolute(x), absolute(xm)))
assert_(eq(np.equal(x, y), equal(xm, ym)))
assert_(eq(np.not_equal(x, y), not_equal(xm, ym)))
assert_(eq(np.less(x, y), less(xm, ym)))
assert_(eq(np.greater(x, y), greater(xm, ym)))
assert_(eq(np.less_equal(x, y), less_equal(xm, ym)))
assert_(eq(np.greater_equal(x, y), greater_equal(xm, ym)))
assert_(eq(np.conjugate(x), conjugate(xm)))
assert_(eq(np.concatenate((x, y)), concatenate((xm, ym))))
assert_(eq(np.concatenate((x, y)), concatenate((x, y))))
assert_(eq(np.concatenate((x, y)), concatenate((xm, y))))
assert_(eq(np.concatenate((x, y, x)), concatenate((x, ym, x))))
def test_xtestCount(self):
# Test count
ott = array([0., 1., 2., 3.], mask=[1, 0, 0, 0])
assert_(count(ott).dtype.type is np.intp)
assert_equal(3, count(ott))
assert_equal(1, count(1))
assert_(eq(0, array(1, mask=[1])))
ott = ott.reshape((2, 2))
assert_(count(ott).dtype.type is np.intp)
assert_(isinstance(count(ott, 0), np.ndarray))
assert_(count(ott).dtype.type is np.intp)
assert_(eq(3, count(ott)))
assert_(getmask(count(ott, 0)) is nomask)
assert_(eq([1, 2], count(ott, 0)))
def test_testMinMax(self):
# Test minimum and maximum.
(x, y, a10, m1, m2, xm, ym, z, zm, xf, s) = self.d
xr = np.ravel(x) # max doesn't work if shaped
xmr = ravel(xm)
# true because of careful selection of data
assert_(eq(max(xr), maximum.reduce(xmr)))
assert_(eq(min(xr), minimum.reduce(xmr)))
def test_testAddSumProd(self):
# Test add, sum, product.
(x, y, a10, m1, m2, xm, ym, z, zm, xf, s) = self.d
assert_(eq(np.add.reduce(x), add.reduce(x)))
assert_(eq(np.add.accumulate(x), add.accumulate(x)))
assert_(eq(4, sum(array(4), axis=0)))
assert_(eq(4, sum(array(4), axis=0)))
assert_(eq(np.sum(x, axis=0), sum(x, axis=0)))
assert_(eq(np.sum(filled(xm, 0), axis=0), sum(xm, axis=0)))
assert_(eq(np.sum(x, 0), sum(x, 0)))
assert_(eq(np.prod(x, axis=0), product(x, axis=0)))
assert_(eq(np.prod(x, 0), product(x, 0)))
assert_(eq(np.prod(filled(xm, 1), axis=0),
product(xm, axis=0)))
if len(s) > 1:
assert_(eq(np.concatenate((x, y), 1),
concatenate((xm, ym), 1)))
assert_(eq(np.add.reduce(x, 1), add.reduce(x, 1)))
assert_(eq(np.sum(x, 1), sum(x, 1)))
assert_(eq(np.prod(x, 1), product(x, 1)))
def test_testCI(self):
# Test of conversions and indexing
x1 = np.array([1, 2, 4, 3])
x2 = array(x1, mask=[1, 0, 0, 0])
x3 = array(x1, mask=[0, 1, 0, 1])
x4 = array(x1)
# test conversion to strings
str(x2) # raises?
repr(x2) # raises?
assert_(eq(np.sort(x1), sort(x2, fill_value=0)))
# tests of indexing
assert_(type(x2[1]) is type(x1[1]))
assert_(x1[1] == x2[1])
assert_(x2[0] is masked)
assert_(eq(x1[2], x2[2]))
assert_(eq(x1[2:5], x2[2:5]))
assert_(eq(x1[:], x2[:]))
assert_(eq(x1[1:], x3[1:]))
x1[2] = 9
x2[2] = 9
assert_(eq(x1, x2))
x1[1:3] = 99
x2[1:3] = 99
assert_(eq(x1, x2))
x2[1] = masked
assert_(eq(x1, x2))
x2[1:3] = masked
assert_(eq(x1, x2))
x2[:] = x1
x2[1] = masked
assert_(allequal(getmask(x2), array([0, 1, 0, 0])))
x3[:] = masked_array([1, 2, 3, 4], [0, 1, 1, 0])
assert_(allequal(getmask(x3), array([0, 1, 1, 0])))
x4[:] = masked_array([1, 2, 3, 4], [0, 1, 1, 0])
assert_(allequal(getmask(x4), array([0, 1, 1, 0])))
assert_(allequal(x4, array([1, 2, 3, 4])))
x1 = np.arange(5) * 1.0
x2 = masked_values(x1, 3.0)
assert_(eq(x1, x2))
assert_(allequal(array([0, 0, 0, 1, 0], MaskType), x2.mask))
assert_(eq(3.0, x2.fill_value))
x1 = array([1, 'hello', 2, 3], object)
x2 = np.array([1, 'hello', 2, 3], object)
s1 = x1[1]
s2 = x2[1]
assert_equal(type(s2), str)
assert_equal(type(s1), str)
assert_equal(s1, s2)
assert_(x1[1:1].shape == (0,))
def test_testCopySize(self):
# Tests of some subtle points of copying and sizing.
n = [0, 0, 1, 0, 0]
m = make_mask(n)
m2 = make_mask(m)
assert_(m is m2)
m3 = make_mask(m, copy=True)
assert_(m is not m3)
x1 = np.arange(5)
y1 = array(x1, mask=m)
assert_(y1._data is not x1)
assert_(allequal(x1, y1._data))
assert_(y1._mask is m)
y1a = array(y1, copy=0)
# For copy=False, one might expect that the array would just
# passed on, i.e., that it would be "is" instead of "==".
# See gh-4043 for discussion.
assert_(y1a._mask.__array_interface__ ==
y1._mask.__array_interface__)
y2 = array(x1, mask=m3, copy=0)
assert_(y2._mask is m3)
assert_(y2[2] is masked)
y2[2] = 9
assert_(y2[2] is not masked)
assert_(y2._mask is m3)
assert_(allequal(y2.mask, 0))
y2a = array(x1, mask=m, copy=1)
assert_(y2a._mask is not m)
assert_(y2a[2] is masked)
y2a[2] = 9
assert_(y2a[2] is not masked)
assert_(y2a._mask is not m)
assert_(allequal(y2a.mask, 0))
y3 = array(x1 * 1.0, mask=m)
assert_(filled(y3).dtype is (x1 * 1.0).dtype)
x4 = arange(4)
x4[2] = masked
y4 = resize(x4, (8,))
assert_(eq(concatenate([x4, x4]), y4))
assert_(eq(getmask(y4), [0, 0, 1, 0, 0, 0, 1, 0]))
y5 = repeat(x4, (2, 2, 2, 2), axis=0)
assert_(eq(y5, [0, 0, 1, 1, 2, 2, 3, 3]))
y6 = repeat(x4, 2, axis=0)
assert_(eq(y5, y6))
def test_testPut(self):
# Test of put
d = arange(5)
n = [0, 0, 0, 1, 1]
m = make_mask(n)
m2 = m.copy()
x = array(d, mask=m)
assert_(x[3] is masked)
assert_(x[4] is masked)
x[[1, 4]] = [10, 40]
assert_(x._mask is m)
assert_(x[3] is masked)
assert_(x[4] is not masked)
assert_(eq(x, [0, 10, 2, -1, 40]))
x = array(d, mask=m2, copy=True)
x.put([0, 1, 2], [-1, 100, 200])
assert_(x._mask is not m2)
assert_(x[3] is masked)
assert_(x[4] is masked)
assert_(eq(x, [-1, 100, 200, 0, 0]))
def test_testPut2(self):
# Test of put
d = arange(5)
x = array(d, mask=[0, 0, 0, 0, 0])
z = array([10, 40], mask=[1, 0])
assert_(x[2] is not masked)
assert_(x[3] is not masked)
x[2:4] = z
assert_(x[2] is masked)
assert_(x[3] is not masked)
assert_(eq(x, [0, 1, 10, 40, 4]))
d = arange(5)
x = array(d, mask=[0, 0, 0, 0, 0])
y = x[2:4]
z = array([10, 40], mask=[1, 0])
assert_(x[2] is not masked)
assert_(x[3] is not masked)
y[:] = z
assert_(y[0] is masked)
assert_(y[1] is not masked)
assert_(eq(y, [10, 40]))
assert_(x[2] is masked)
assert_(x[3] is not masked)
assert_(eq(x, [0, 1, 10, 40, 4]))
def test_testMaPut(self):
(x, y, a10, m1, m2, xm, ym, z, zm, xf, s) = self.d
m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1]
i = np.nonzero(m)[0]
put(ym, i, zm)
assert_(all(take(ym, i, axis=0) == zm))
def test_testOddFeatures(self):
# Test of other odd features
x = arange(20)
x = x.reshape(4, 5)
x.flat[5] = 12
assert_(x[1, 0] == 12)
z = x + 10j * x
assert_(eq(z.real, x))
assert_(eq(z.imag, 10 * x))
assert_(eq((z * conjugate(z)).real, 101 * x * x))
z.imag[...] = 0.0
x = arange(10)
x[3] = masked
assert_(str(x[3]) == str(masked))
c = x >= 8
assert_(count(where(c, masked, masked)) == 0)
assert_(shape(where(c, masked, masked)) == c.shape)
z = where(c, x, masked)
assert_(z.dtype is x.dtype)
assert_(z[3] is masked)
assert_(z[4] is masked)
assert_(z[7] is masked)
assert_(z[8] is not masked)
assert_(z[9] is not masked)
assert_(eq(x, z))
z = where(c, masked, x)
assert_(z.dtype is x.dtype)
assert_(z[3] is masked)
assert_(z[4] is not masked)
assert_(z[7] is not masked)
assert_(z[8] is masked)
assert_(z[9] is masked)
z = masked_where(c, x)
assert_(z.dtype is x.dtype)
assert_(z[3] is masked)
assert_(z[4] is not masked)
assert_(z[7] is not masked)
assert_(z[8] is masked)
assert_(z[9] is masked)
assert_(eq(x, z))
x = array([1., 2., 3., 4., 5.])
c = array([1, 1, 1, 0, 0])
x[2] = masked
z = where(c, x, -x)
assert_(eq(z, [1., 2., 0., -4., -5]))
c[0] = masked
z = where(c, x, -x)
assert_(eq(z, [1., 2., 0., -4., -5]))
assert_(z[0] is masked)
assert_(z[1] is not masked)
assert_(z[2] is masked)
assert_(eq(masked_where(greater(x, 2), x), masked_greater(x, 2)))
assert_(eq(masked_where(greater_equal(x, 2), x),
masked_greater_equal(x, 2)))
assert_(eq(masked_where(less(x, 2), x), masked_less(x, 2)))
assert_(eq(masked_where(less_equal(x, 2), x), masked_less_equal(x, 2)))
assert_(eq(masked_where(not_equal(x, 2), x), masked_not_equal(x, 2)))
assert_(eq(masked_where(equal(x, 2), x), masked_equal(x, 2)))
assert_(eq(masked_where(not_equal(x, 2), x), masked_not_equal(x, 2)))
assert_(eq(masked_inside(list(range(5)), 1, 3), [0, 199, 199, 199, 4]))
assert_(eq(masked_outside(list(range(5)), 1, 3), [199, 1, 2, 3, 199]))
assert_(eq(masked_inside(array(list(range(5)),
mask=[1, 0, 0, 0, 0]), 1, 3).mask,
[1, 1, 1, 1, 0]))
assert_(eq(masked_outside(array(list(range(5)),
mask=[0, 1, 0, 0, 0]), 1, 3).mask,
[1, 1, 0, 0, 1]))
assert_(eq(masked_equal(array(list(range(5)),
mask=[1, 0, 0, 0, 0]), 2).mask,
[1, 0, 1, 0, 0]))
assert_(eq(masked_not_equal(array([2, 2, 1, 2, 1],
mask=[1, 0, 0, 0, 0]), 2).mask,
[1, 0, 1, 0, 1]))
assert_(eq(masked_where([1, 1, 0, 0, 0], [1, 2, 3, 4, 5]),
[99, 99, 3, 4, 5]))
atest = ones((10, 10, 10), dtype=np.float32)
btest = zeros(atest.shape, MaskType)
ctest = masked_where(btest, atest)
assert_(eq(atest, ctest))
z = choose(c, (-x, x))
assert_(eq(z, [1., 2., 0., -4., -5]))
assert_(z[0] is masked)
assert_(z[1] is not masked)
assert_(z[2] is masked)
x = arange(6)
x[5] = masked
y = arange(6) * 10
y[2] = masked
c = array([1, 1, 1, 0, 0, 0], mask=[1, 0, 0, 0, 0, 0])
cm = c.filled(1)
z = where(c, x, y)
zm = where(cm, x, y)
assert_(eq(z, zm))
assert_(getmask(zm) is nomask)
assert_(eq(zm, [0, 1, 2, 30, 40, 50]))
z = where(c, masked, 1)
assert_(eq(z, [99, 99, 99, 1, 1, 1]))
z = where(c, 1, masked)
assert_(eq(z, [99, 1, 1, 99, 99, 99]))
def test_testMinMax2(self):
# Test of minimum, maximum.
assert_(eq(minimum([1, 2, 3], [4, 0, 9]), [1, 0, 3]))
assert_(eq(maximum([1, 2, 3], [4, 0, 9]), [4, 2, 9]))
x = arange(5)
y = arange(5) - 2
x[3] = masked
y[0] = masked
assert_(eq(minimum(x, y), where(less(x, y), x, y)))
assert_(eq(maximum(x, y), where(greater(x, y), x, y)))
assert_(minimum.reduce(x) == 0)
assert_(maximum.reduce(x) == 4)
def test_testTakeTransposeInnerOuter(self):
# Test of take, transpose, inner, outer products
x = arange(24)
y = np.arange(24)
x[5:6] = masked
x = x.reshape(2, 3, 4)
y = y.reshape(2, 3, 4)
assert_(eq(np.transpose(y, (2, 0, 1)), transpose(x, (2, 0, 1))))
assert_(eq(np.take(y, (2, 0, 1), 1), take(x, (2, 0, 1), 1)))
assert_(eq(np.inner(filled(x, 0), filled(y, 0)),
inner(x, y)))
assert_(eq(np.outer(filled(x, 0), filled(y, 0)),
outer(x, y)))
y = array(['abc', 1, 'def', 2, 3], object)
y[2] = masked
t = take(y, [0, 3, 4])
assert_(t[0] == 'abc')
assert_(t[1] == 2)
assert_(t[2] == 3)
def test_testInplace(self):
# Test of inplace operations and rich comparisons
y = arange(10)
x = arange(10)
xm = arange(10)
xm[2] = masked
x += 1
assert_(eq(x, y + 1))
xm += 1
assert_(eq(x, y + 1))
x = arange(10)
xm = arange(10)
xm[2] = masked
x -= 1
assert_(eq(x, y - 1))
xm -= 1
assert_(eq(xm, y - 1))
x = arange(10) * 1.0
xm = arange(10) * 1.0
xm[2] = masked
x *= 2.0
assert_(eq(x, y * 2))
xm *= 2.0
assert_(eq(xm, y * 2))
x = arange(10) * 2
xm = arange(10)
xm[2] = masked
x //= 2
assert_(eq(x, y))
xm //= 2
assert_(eq(x, y))
x = arange(10) * 1.0
xm = arange(10) * 1.0
xm[2] = masked
x /= 2.0
assert_(eq(x, y / 2.0))
xm /= arange(10)
assert_(eq(xm, ones((10,))))
x = arange(10).astype(np.float32)
xm = arange(10)
xm[2] = masked
x += 1.
assert_(eq(x, y + 1.))
def test_testPickle(self):
# Test of pickling
x = arange(12)
x[4:10:2] = masked
x = x.reshape(4, 3)
for proto in range(2, pickle.HIGHEST_PROTOCOL + 1):
s = pickle.dumps(x, protocol=proto)
y = pickle.loads(s)
assert_(eq(x, y))
def test_testMasked(self):
# Test of masked element
xx = arange(6)
xx[1] = masked
assert_(str(masked) == '--')
assert_(xx[1] is masked)
assert_equal(filled(xx[1], 0), 0)
def test_testAverage1(self):
# Test of average.
ott = array([0., 1., 2., 3.], mask=[1, 0, 0, 0])
assert_(eq(2.0, average(ott, axis=0)))
assert_(eq(2.0, average(ott, weights=[1., 1., 2., 1.])))
result, wts = average(ott, weights=[1., 1., 2., 1.], returned=True)
assert_(eq(2.0, result))
assert_(wts == 4.0)
ott[:] = masked
assert_(average(ott, axis=0) is masked)
ott = array([0., 1., 2., 3.], mask=[1, 0, 0, 0])
ott = ott.reshape(2, 2)
ott[:, 1] = masked
assert_(eq(average(ott, axis=0), [2.0, 0.0]))
assert_(average(ott, axis=1)[0] is masked)
assert_(eq([2., 0.], average(ott, axis=0)))
result, wts = average(ott, axis=0, returned=True)
assert_(eq(wts, [1., 0.]))
def test_testAverage2(self):
# More tests of average.
w1 = [0, 1, 1, 1, 1, 0]
w2 = [[0, 1, 1, 1, 1, 0], [1, 0, 0, 0, 0, 1]]
x = arange(6)
assert_(allclose(average(x, axis=0), 2.5))
assert_(allclose(average(x, axis=0, weights=w1), 2.5))
y = array([arange(6), 2.0 * arange(6)])
assert_(allclose(average(y, None),
np.add.reduce(np.arange(6)) * 3. / 12.))
assert_(allclose(average(y, axis=0), np.arange(6) * 3. / 2.))
assert_(allclose(average(y, axis=1),
[average(x, axis=0), average(x, axis=0) * 2.0]))
assert_(allclose(average(y, None, weights=w2), 20. / 6.))
assert_(allclose(average(y, axis=0, weights=w2),
[0., 1., 2., 3., 4., 10.]))
assert_(allclose(average(y, axis=1),
[average(x, axis=0), average(x, axis=0) * 2.0]))
m1 = zeros(6)
m2 = [0, 0, 1, 1, 0, 0]
m3 = [[0, 0, 1, 1, 0, 0], [0, 1, 1, 1, 1, 0]]
m4 = ones(6)
m5 = [0, 1, 1, 1, 1, 1]
assert_(allclose(average(masked_array(x, m1), axis=0), 2.5))
assert_(allclose(average(masked_array(x, m2), axis=0), 2.5))
assert_(average(masked_array(x, m4), axis=0) is masked)
assert_equal(average(masked_array(x, m5), axis=0), 0.0)
assert_equal(count(average(masked_array(x, m4), axis=0)), 0)
z = masked_array(y, m3)
assert_(allclose(average(z, None), 20. / 6.))
assert_(allclose(average(z, axis=0),
[0., 1., 99., 99., 4.0, 7.5]))
assert_(allclose(average(z, axis=1), [2.5, 5.0]))
assert_(allclose(average(z, axis=0, weights=w2),
[0., 1., 99., 99., 4.0, 10.0]))
a = arange(6)
b = arange(6) * 3
r1, w1 = average([[a, b], [b, a]], axis=1, returned=True)
assert_equal(shape(r1), shape(w1))
assert_equal(r1.shape, w1.shape)
r2, w2 = average(ones((2, 2, 3)), axis=0, weights=[3, 1], returned=True)
assert_equal(shape(w2), shape(r2))
r2, w2 = average(ones((2, 2, 3)), returned=True)
assert_equal(shape(w2), shape(r2))
r2, w2 = average(ones((2, 2, 3)), weights=ones((2, 2, 3)), returned=True)
assert_(shape(w2) == shape(r2))
a2d = array([[1, 2], [0, 4]], float)
a2dm = masked_array(a2d, [[0, 0], [1, 0]])
a2da = average(a2d, axis=0)
assert_(eq(a2da, [0.5, 3.0]))
a2dma = average(a2dm, axis=0)
assert_(eq(a2dma, [1.0, 3.0]))
a2dma = average(a2dm, axis=None)
assert_(eq(a2dma, 7. / 3.))
a2dma = average(a2dm, axis=1)
assert_(eq(a2dma, [1.5, 4.0]))
def test_testToPython(self):
assert_equal(1, int(array(1)))
assert_equal(1.0, float(array(1)))
assert_equal(1, int(array([[[1]]])))
assert_equal(1.0, float(array([[1]])))
assert_raises(TypeError, float, array([1, 1]))
assert_raises(ValueError, bool, array([0, 1]))
assert_raises(ValueError, bool, array([0, 0], mask=[0, 1]))
def test_testScalarArithmetic(self):
xm = array(0, mask=1)
# TODO FIXME: Find out what the following raises a warning in r8247
with np.errstate(divide='ignore'):
assert_((1 / array(0)).mask)
assert_((1 + xm).mask)
assert_((-xm).mask)
assert_((-xm).mask)
assert_(maximum(xm, xm).mask)
assert_(minimum(xm, xm).mask)
assert_(xm.filled().dtype is xm._data.dtype)
x = array(0, mask=0)
assert_(x.filled() == x._data)
assert_equal(str(xm), str(masked_print_option))
def test_testArrayMethods(self):
a = array([1, 3, 2])
assert_(eq(a.any(), a._data.any()))
assert_(eq(a.all(), a._data.all()))
assert_(eq(a.argmax(), a._data.argmax()))
assert_(eq(a.argmin(), a._data.argmin()))
assert_(eq(a.choose(0, 1, 2, 3, 4),
a._data.choose(0, 1, 2, 3, 4)))
assert_(eq(a.compress([1, 0, 1]), a._data.compress([1, 0, 1])))
assert_(eq(a.conj(), a._data.conj()))
assert_(eq(a.conjugate(), a._data.conjugate()))
m = array([[1, 2], [3, 4]])
assert_(eq(m.diagonal(), m._data.diagonal()))
assert_(eq(a.sum(), a._data.sum()))
assert_(eq(a.take([1, 2]), a._data.take([1, 2])))
assert_(eq(m.transpose(), m._data.transpose()))
def test_testArrayAttributes(self):
a = array([1, 3, 2])
assert_equal(a.ndim, 1)
def test_testAPI(self):
assert_(not [m for m in dir(np.ndarray)
if m not in dir(MaskedArray) and
not m.startswith('_')])
def test_testSingleElementSubscript(self):
a = array([1, 3, 2])
b = array([1, 3, 2], mask=[1, 0, 1])
assert_equal(a[0].shape, ())
assert_equal(b[0].shape, ())
assert_equal(b[1].shape, ())
def test_assignment_by_condition(self):
# Test for gh-18951
a = array([1, 2, 3, 4], mask=[1, 0, 1, 0])
c = a >= 3
a[c] = 5
assert_(a[2] is masked)
def test_assignment_by_condition_2(self):
# gh-19721
a = masked_array([0, 1], mask=[False, False])
b = masked_array([0, 1], mask=[True, True])
mask = a < 1
b[mask] = a[mask]
expected_mask = [False, True]
assert_equal(b.mask, expected_mask)
class TestUfuncs:
def setup_method(self):
self.d = (array([1.0, 0, -1, pi / 2] * 2, mask=[0, 1] + [0] * 6),
array([1.0, 0, -1, pi / 2] * 2, mask=[1, 0] + [0] * 6),)
def test_testUfuncRegression(self):
f_invalid_ignore = [
'sqrt', 'arctanh', 'arcsin', 'arccos',
'arccosh', 'arctanh', 'log', 'log10', 'divide',
'true_divide', 'floor_divide', 'remainder', 'fmod']
for f in ['sqrt', 'log', 'log10', 'exp', 'conjugate',
'sin', 'cos', 'tan',
'arcsin', 'arccos', 'arctan',
'sinh', 'cosh', 'tanh',
'arcsinh',
'arccosh',
'arctanh',
'absolute', 'fabs', 'negative',
'floor', 'ceil',
'logical_not',
'add', 'subtract', 'multiply',
'divide', 'true_divide', 'floor_divide',
'remainder', 'fmod', 'hypot', 'arctan2',
'equal', 'not_equal', 'less_equal', 'greater_equal',
'less', 'greater',
'logical_and', 'logical_or', 'logical_xor']:
try:
uf = getattr(umath, f)
except AttributeError:
uf = getattr(fromnumeric, f)
mf = getattr(np.ma, f)
args = self.d[:uf.nin]
with np.errstate():
if f in f_invalid_ignore:
np.seterr(invalid='ignore')
if f in ['arctanh', 'log', 'log10']:
np.seterr(divide='ignore')
ur = uf(*args)
mr = mf(*args)
assert_(eq(ur.filled(0), mr.filled(0), f))
assert_(eqmask(ur.mask, mr.mask))
def test_reduce(self):
a = self.d[0]
assert_(not alltrue(a, axis=0))
assert_(sometrue(a, axis=0))
assert_equal(sum(a[:3], axis=0), 0)
assert_equal(product(a, axis=0), 0)
def test_minmax(self):
a = arange(1, 13).reshape(3, 4)
amask = masked_where(a < 5, a)
assert_equal(amask.max(), a.max())
assert_equal(amask.min(), 5)
assert_((amask.max(0) == a.max(0)).all())
assert_((amask.min(0) == [5, 6, 7, 8]).all())
assert_(amask.max(1)[0].mask)
assert_(amask.min(1)[0].mask)
def test_nonzero(self):
for t in "?bhilqpBHILQPfdgFDGO":
x = array([1, 0, 2, 0], mask=[0, 0, 1, 1])
assert_(eq(nonzero(x), [0]))
class TestArrayMethods:
def setup_method(self):
x = np.array([8.375, 7.545, 8.828, 8.5, 1.757, 5.928,
8.43, 7.78, 9.865, 5.878, 8.979, 4.732,
3.012, 6.022, 5.095, 3.116, 5.238, 3.957,
6.04, 9.63, 7.712, 3.382, 4.489, 6.479,
7.189, 9.645, 5.395, 4.961, 9.894, 2.893,
7.357, 9.828, 6.272, 3.758, 6.693, 0.993])
X = x.reshape(6, 6)
XX = x.reshape(3, 2, 2, 3)
m = np.array([0, 1, 0, 1, 0, 0,
1, 0, 1, 1, 0, 1,
0, 0, 0, 1, 0, 1,
0, 0, 0, 1, 1, 1,
1, 0, 0, 1, 0, 0,
0, 0, 1, 0, 1, 0])
mx = array(data=x, mask=m)
mX = array(data=X, mask=m.reshape(X.shape))
mXX = array(data=XX, mask=m.reshape(XX.shape))
self.d = (x, X, XX, m, mx, mX, mXX)
def test_trace(self):
(x, X, XX, m, mx, mX, mXX,) = self.d
mXdiag = mX.diagonal()
assert_equal(mX.trace(), mX.diagonal().compressed().sum())
assert_(eq(mX.trace(),
X.trace() - sum(mXdiag.mask * X.diagonal(),
axis=0)))
def test_clip(self):
(x, X, XX, m, mx, mX, mXX,) = self.d
clipped = mx.clip(2, 8)
assert_(eq(clipped.mask, mx.mask))
assert_(eq(clipped._data, x.clip(2, 8)))
assert_(eq(clipped._data, mx._data.clip(2, 8)))
def test_ptp(self):
(x, X, XX, m, mx, mX, mXX,) = self.d
(n, m) = X.shape
# print(type(mx), mx.compressed())
# raise Exception()
assert_equal(mx.ptp(), np.ptp(mx.compressed()))
rows = np.zeros(n, np.float64)
cols = np.zeros(m, np.float64)
for k in range(m):
cols[k] = np.ptp(mX[:, k].compressed())
for k in range(n):
rows[k] = np.ptp(mX[k].compressed())
assert_(eq(mX.ptp(0), cols))
assert_(eq(mX.ptp(1), rows))
def test_swapaxes(self):
(x, X, XX, m, mx, mX, mXX,) = self.d
mXswapped = mX.swapaxes(0, 1)
assert_(eq(mXswapped[-1], mX[:, -1]))
mXXswapped = mXX.swapaxes(0, 2)
assert_equal(mXXswapped.shape, (2, 2, 3, 3))
def test_cumprod(self):
(x, X, XX, m, mx, mX, mXX,) = self.d
mXcp = mX.cumprod(0)
assert_(eq(mXcp._data, mX.filled(1).cumprod(0)))
mXcp = mX.cumprod(1)
assert_(eq(mXcp._data, mX.filled(1).cumprod(1)))
def test_cumsum(self):
(x, X, XX, m, mx, mX, mXX,) = self.d
mXcp = mX.cumsum(0)
assert_(eq(mXcp._data, mX.filled(0).cumsum(0)))
mXcp = mX.cumsum(1)
assert_(eq(mXcp._data, mX.filled(0).cumsum(1)))
def test_varstd(self):
(x, X, XX, m, mx, mX, mXX,) = self.d
assert_(eq(mX.var(axis=None), mX.compressed().var()))
assert_(eq(mX.std(axis=None), mX.compressed().std()))
assert_(eq(mXX.var(axis=3).shape, XX.var(axis=3).shape))
assert_(eq(mX.var().shape, X.var().shape))
(mXvar0, mXvar1) = (mX.var(axis=0), mX.var(axis=1))
for k in range(6):
assert_(eq(mXvar1[k], mX[k].compressed().var()))
assert_(eq(mXvar0[k], mX[:, k].compressed().var()))
assert_(eq(np.sqrt(mXvar0[k]),
mX[:, k].compressed().std()))
def eqmask(m1, m2):
if m1 is nomask:
return m2 is nomask
if m2 is nomask:
return m1 is nomask
return (m1 == m2).all()

View File

@ -0,0 +1,100 @@
import numpy as np
from numpy.testing import (
assert_,
assert_allclose,
assert_array_equal,
suppress_warnings,
)
class TestRegression:
def test_masked_array_create(self):
# Ticket #17
x = np.ma.masked_array([0, 1, 2, 3, 0, 4, 5, 6],
mask=[0, 0, 0, 1, 1, 1, 0, 0])
assert_array_equal(np.ma.nonzero(x), [[1, 2, 6, 7]])
def test_masked_array(self):
# Ticket #61
np.ma.array(1, mask=[1])
def test_mem_masked_where(self):
# Ticket #62
from numpy.ma import MaskType, masked_where
a = np.zeros((1, 1))
b = np.zeros(a.shape, MaskType)
c = masked_where(b, a)
a - c
def test_masked_array_multiply(self):
# Ticket #254
a = np.ma.zeros((4, 1))
a[2, 0] = np.ma.masked
b = np.zeros((4, 2))
a * b
b * a
def test_masked_array_repeat(self):
# Ticket #271
np.ma.array([1], mask=False).repeat(10)
def test_masked_array_repr_unicode(self):
# Ticket #1256
repr(np.ma.array("Unicode"))
def test_atleast_2d(self):
# Ticket #1559
a = np.ma.masked_array([0.0, 1.2, 3.5], mask=[False, True, False])
b = np.atleast_2d(a)
assert_(a.mask.ndim == 1)
assert_(b.mask.ndim == 2)
def test_set_fill_value_unicode_py3(self):
# Ticket #2733
a = np.ma.masked_array(['a', 'b', 'c'], mask=[1, 0, 0])
a.fill_value = 'X'
assert_(a.fill_value == 'X')
def test_var_sets_maskedarray_scalar(self):
# Issue gh-2757
a = np.ma.array(np.arange(5), mask=True)
mout = np.ma.array(-1, dtype=float)
a.var(out=mout)
assert_(mout._data == 0)
def test_ddof_corrcoef(self):
# See gh-3336
x = np.ma.masked_equal([1, 2, 3, 4, 5], 4)
y = np.array([2, 2.5, 3.1, 3, 5])
# this test can be removed after deprecation.
with suppress_warnings() as sup:
sup.filter(DeprecationWarning, "bias and ddof have no effect")
r0 = np.ma.corrcoef(x, y, ddof=0)
r1 = np.ma.corrcoef(x, y, ddof=1)
# ddof should not have an effect (it gets cancelled out)
assert_allclose(r0.data, r1.data)
def test_mask_not_backmangled(self):
# See gh-10314. Test case taken from gh-3140.
a = np.ma.MaskedArray([1., 2.], mask=[False, False])
assert_(a.mask.shape == (2,))
b = np.tile(a, (2, 1))
# Check that the above no longer changes a.shape to (1, 2)
assert_(a.mask.shape == (2,))
assert_(b.shape == (2, 2))
assert_(b.mask.shape == (2, 2))
def test_empty_list_on_structured(self):
# See gh-12464. Indexing with empty list should give empty result.
ma = np.ma.MaskedArray([(1, 1.), (2, 2.), (3, 3.)], dtype='i4,f4')
assert_array_equal(ma[[]], ma[:0])
def test_masked_array_tobytes_fortran(self):
ma = np.ma.arange(4).reshape((2, 2))
assert_array_equal(ma.tobytes(order='F'), ma.T.tobytes())
def test_structured_array(self):
# see gh-22041
np.ma.array((1, (b"", b"")),
dtype=[("x", np.int_),
("y", [("i", np.void), ("j", np.void)])])

View File

@ -0,0 +1,469 @@
"""Tests suite for MaskedArray & subclassing.
:author: Pierre Gerard-Marchant
:contact: pierregm_at_uga_dot_edu
"""
import numpy as np
from numpy.lib.mixins import NDArrayOperatorsMixin
from numpy.ma.core import (
MaskedArray,
add,
arange,
array,
asanyarray,
asarray,
divide,
hypot,
log,
masked,
masked_array,
nomask,
)
from numpy.ma.testutils import assert_equal
from numpy.testing import assert_, assert_raises
# from numpy.ma.core import (
def assert_startswith(a, b):
# produces a better error message than assert_(a.startswith(b))
assert_equal(a[:len(b)], b)
class SubArray(np.ndarray):
# Defines a generic np.ndarray subclass, that stores some metadata
# in the dictionary `info`.
def __new__(cls, arr, info={}):
x = np.asanyarray(arr).view(cls)
x.info = info.copy()
return x
def __array_finalize__(self, obj):
super().__array_finalize__(obj)
self.info = getattr(obj, 'info', {}).copy()
def __add__(self, other):
result = super().__add__(other)
result.info['added'] = result.info.get('added', 0) + 1
return result
def __iadd__(self, other):
result = super().__iadd__(other)
result.info['iadded'] = result.info.get('iadded', 0) + 1
return result
subarray = SubArray
class SubMaskedArray(MaskedArray):
"""Pure subclass of MaskedArray, keeping some info on subclass."""
def __new__(cls, info=None, **kwargs):
obj = super().__new__(cls, **kwargs)
obj._optinfo['info'] = info
return obj
class MSubArray(SubArray, MaskedArray):
def __new__(cls, data, info={}, mask=nomask):
subarr = SubArray(data, info)
_data = MaskedArray.__new__(cls, data=subarr, mask=mask)
_data.info = subarr.info
return _data
@property
def _series(self):
_view = self.view(MaskedArray)
_view._sharedmask = False
return _view
msubarray = MSubArray
# Also a subclass that overrides __str__, __repr__ and __setitem__, disallowing
# setting to non-class values (and thus np.ma.core.masked_print_option)
# and overrides __array_wrap__, updating the info dict, to check that this
# doesn't get destroyed by MaskedArray._update_from. But this one also needs
# its own iterator...
class CSAIterator:
"""
Flat iterator object that uses its own setter/getter
(works around ndarray.flat not propagating subclass setters/getters
see https://github.com/numpy/numpy/issues/4564)
roughly following MaskedIterator
"""
def __init__(self, a):
self._original = a
self._dataiter = a.view(np.ndarray).flat
def __iter__(self):
return self
def __getitem__(self, indx):
out = self._dataiter.__getitem__(indx)
if not isinstance(out, np.ndarray):
out = out.__array__()
out = out.view(type(self._original))
return out
def __setitem__(self, index, value):
self._dataiter[index] = self._original._validate_input(value)
def __next__(self):
return next(self._dataiter).__array__().view(type(self._original))
class ComplicatedSubArray(SubArray):
def __str__(self):
return f'myprefix {self.view(SubArray)} mypostfix'
def __repr__(self):
# Return a repr that does not start with 'name('
return f'<{self.__class__.__name__} {self}>'
def _validate_input(self, value):
if not isinstance(value, ComplicatedSubArray):
raise ValueError("Can only set to MySubArray values")
return value
def __setitem__(self, item, value):
# validation ensures direct assignment with ndarray or
# masked_print_option will fail
super().__setitem__(item, self._validate_input(value))
def __getitem__(self, item):
# ensure getter returns our own class also for scalars
value = super().__getitem__(item)
if not isinstance(value, np.ndarray): # scalar
value = value.__array__().view(ComplicatedSubArray)
return value
@property
def flat(self):
return CSAIterator(self)
@flat.setter
def flat(self, value):
y = self.ravel()
y[:] = value
def __array_wrap__(self, obj, context=None, return_scalar=False):
obj = super().__array_wrap__(obj, context, return_scalar)
if context is not None and context[0] is np.multiply:
obj.info['multiplied'] = obj.info.get('multiplied', 0) + 1
return obj
class WrappedArray(NDArrayOperatorsMixin):
"""
Wrapping a MaskedArray rather than subclassing to test that
ufunc deferrals are commutative.
See: https://github.com/numpy/numpy/issues/15200)
"""
__slots__ = ('_array', 'attrs')
__array_priority__ = 20
def __init__(self, array, **attrs):
self._array = array
self.attrs = attrs
def __repr__(self):
return f"{self.__class__.__name__}(\n{self._array}\n{self.attrs}\n)"
def __array__(self, dtype=None, copy=None):
return np.asarray(self._array)
def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
if method == '__call__':
inputs = [arg._array if isinstance(arg, self.__class__) else arg
for arg in inputs]
return self.__class__(ufunc(*inputs, **kwargs), **self.attrs)
else:
return NotImplemented
class TestSubclassing:
# Test suite for masked subclasses of ndarray.
def setup_method(self):
x = np.arange(5, dtype='float')
mx = msubarray(x, mask=[0, 1, 0, 0, 0])
self.data = (x, mx)
def test_data_subclassing(self):
# Tests whether the subclass is kept.
x = np.arange(5)
m = [0, 0, 1, 0, 0]
xsub = SubArray(x)
xmsub = masked_array(xsub, mask=m)
assert_(isinstance(xmsub, MaskedArray))
assert_equal(xmsub._data, xsub)
assert_(isinstance(xmsub._data, SubArray))
def test_maskedarray_subclassing(self):
# Tests subclassing MaskedArray
(x, mx) = self.data
assert_(isinstance(mx._data, subarray))
def test_masked_unary_operations(self):
# Tests masked_unary_operation
(x, mx) = self.data
with np.errstate(divide='ignore'):
assert_(isinstance(log(mx), msubarray))
assert_equal(log(x), np.log(x))
def test_masked_binary_operations(self):
# Tests masked_binary_operation
(x, mx) = self.data
# Result should be a msubarray
assert_(isinstance(add(mx, mx), msubarray))
assert_(isinstance(add(mx, x), msubarray))
# Result should work
assert_equal(add(mx, x), mx + x)
assert_(isinstance(add(mx, mx)._data, subarray))
assert_(isinstance(add.outer(mx, mx), msubarray))
assert_(isinstance(hypot(mx, mx), msubarray))
assert_(isinstance(hypot(mx, x), msubarray))
def test_masked_binary_operations2(self):
# Tests domained_masked_binary_operation
(x, mx) = self.data
xmx = masked_array(mx.data.__array__(), mask=mx.mask)
assert_(isinstance(divide(mx, mx), msubarray))
assert_(isinstance(divide(mx, x), msubarray))
assert_equal(divide(mx, mx), divide(xmx, xmx))
def test_attributepropagation(self):
x = array(arange(5), mask=[0] + [1] * 4)
my = masked_array(subarray(x))
ym = msubarray(x)
#
z = (my + 1)
assert_(isinstance(z, MaskedArray))
assert_(not isinstance(z, MSubArray))
assert_(isinstance(z._data, SubArray))
assert_equal(z._data.info, {})
#
z = (ym + 1)
assert_(isinstance(z, MaskedArray))
assert_(isinstance(z, MSubArray))
assert_(isinstance(z._data, SubArray))
assert_(z._data.info['added'] > 0)
# Test that inplace methods from data get used (gh-4617)
ym += 1
assert_(isinstance(ym, MaskedArray))
assert_(isinstance(ym, MSubArray))
assert_(isinstance(ym._data, SubArray))
assert_(ym._data.info['iadded'] > 0)
#
ym._set_mask([1, 0, 0, 0, 1])
assert_equal(ym._mask, [1, 0, 0, 0, 1])
ym._series._set_mask([0, 0, 0, 0, 1])
assert_equal(ym._mask, [0, 0, 0, 0, 1])
#
xsub = subarray(x, info={'name': 'x'})
mxsub = masked_array(xsub)
assert_(hasattr(mxsub, 'info'))
assert_equal(mxsub.info, xsub.info)
def test_subclasspreservation(self):
# Checks that masked_array(...,subok=True) preserves the class.
x = np.arange(5)
m = [0, 0, 1, 0, 0]
xinfo = list(zip(x, m))
xsub = MSubArray(x, mask=m, info={'xsub': xinfo})
#
mxsub = masked_array(xsub, subok=False)
assert_(not isinstance(mxsub, MSubArray))
assert_(isinstance(mxsub, MaskedArray))
assert_equal(mxsub._mask, m)
#
mxsub = asarray(xsub)
assert_(not isinstance(mxsub, MSubArray))
assert_(isinstance(mxsub, MaskedArray))
assert_equal(mxsub._mask, m)
#
mxsub = masked_array(xsub, subok=True)
assert_(isinstance(mxsub, MSubArray))
assert_equal(mxsub.info, xsub.info)
assert_equal(mxsub._mask, xsub._mask)
#
mxsub = asanyarray(xsub)
assert_(isinstance(mxsub, MSubArray))
assert_equal(mxsub.info, xsub.info)
assert_equal(mxsub._mask, m)
def test_subclass_items(self):
"""test that getter and setter go via baseclass"""
x = np.arange(5)
xcsub = ComplicatedSubArray(x)
mxcsub = masked_array(xcsub, mask=[True, False, True, False, False])
# getter should return a ComplicatedSubArray, even for single item
# first check we wrote ComplicatedSubArray correctly
assert_(isinstance(xcsub[1], ComplicatedSubArray))
assert_(isinstance(xcsub[1, ...], ComplicatedSubArray))
assert_(isinstance(xcsub[1:4], ComplicatedSubArray))
# now that it propagates inside the MaskedArray
assert_(isinstance(mxcsub[1], ComplicatedSubArray))
assert_(isinstance(mxcsub[1, ...].data, ComplicatedSubArray))
assert_(mxcsub[0] is masked)
assert_(isinstance(mxcsub[0, ...].data, ComplicatedSubArray))
assert_(isinstance(mxcsub[1:4].data, ComplicatedSubArray))
# also for flattened version (which goes via MaskedIterator)
assert_(isinstance(mxcsub.flat[1].data, ComplicatedSubArray))
assert_(mxcsub.flat[0] is masked)
assert_(isinstance(mxcsub.flat[1:4].base, ComplicatedSubArray))
# setter should only work with ComplicatedSubArray input
# first check we wrote ComplicatedSubArray correctly
assert_raises(ValueError, xcsub.__setitem__, 1, x[4])
# now that it propagates inside the MaskedArray
assert_raises(ValueError, mxcsub.__setitem__, 1, x[4])
assert_raises(ValueError, mxcsub.__setitem__, slice(1, 4), x[1:4])
mxcsub[1] = xcsub[4]
mxcsub[1:4] = xcsub[1:4]
# also for flattened version (which goes via MaskedIterator)
assert_raises(ValueError, mxcsub.flat.__setitem__, 1, x[4])
assert_raises(ValueError, mxcsub.flat.__setitem__, slice(1, 4), x[1:4])
mxcsub.flat[1] = xcsub[4]
mxcsub.flat[1:4] = xcsub[1:4]
def test_subclass_nomask_items(self):
x = np.arange(5)
xcsub = ComplicatedSubArray(x)
mxcsub_nomask = masked_array(xcsub)
assert_(isinstance(mxcsub_nomask[1, ...].data, ComplicatedSubArray))
assert_(isinstance(mxcsub_nomask[0, ...].data, ComplicatedSubArray))
assert_(isinstance(mxcsub_nomask[1], ComplicatedSubArray))
assert_(isinstance(mxcsub_nomask[0], ComplicatedSubArray))
def test_subclass_repr(self):
"""test that repr uses the name of the subclass
and 'array' for np.ndarray"""
x = np.arange(5)
mx = masked_array(x, mask=[True, False, True, False, False])
assert_startswith(repr(mx), 'masked_array')
xsub = SubArray(x)
mxsub = masked_array(xsub, mask=[True, False, True, False, False])
assert_startswith(repr(mxsub),
f'masked_{SubArray.__name__}(data=[--, 1, --, 3, 4]')
def test_subclass_str(self):
"""test str with subclass that has overridden str, setitem"""
# first without override
x = np.arange(5)
xsub = SubArray(x)
mxsub = masked_array(xsub, mask=[True, False, True, False, False])
assert_equal(str(mxsub), '[-- 1 -- 3 4]')
xcsub = ComplicatedSubArray(x)
assert_raises(ValueError, xcsub.__setitem__, 0,
np.ma.core.masked_print_option)
mxcsub = masked_array(xcsub, mask=[True, False, True, False, False])
assert_equal(str(mxcsub), 'myprefix [-- 1 -- 3 4] mypostfix')
def test_pure_subclass_info_preservation(self):
# Test that ufuncs and methods conserve extra information consistently;
# see gh-7122.
arr1 = SubMaskedArray('test', data=[1, 2, 3, 4, 5, 6])
arr2 = SubMaskedArray(data=[0, 1, 2, 3, 4, 5])
diff1 = np.subtract(arr1, arr2)
assert_('info' in diff1._optinfo)
assert_(diff1._optinfo['info'] == 'test')
diff2 = arr1 - arr2
assert_('info' in diff2._optinfo)
assert_(diff2._optinfo['info'] == 'test')
class ArrayNoInheritance:
"""Quantity-like class that does not inherit from ndarray"""
def __init__(self, data, units):
self.magnitude = data
self.units = units
def __getattr__(self, attr):
return getattr(self.magnitude, attr)
def test_array_no_inheritance():
data_masked = np.ma.array([1, 2, 3], mask=[True, False, True])
data_masked_units = ArrayNoInheritance(data_masked, 'meters')
# Get the masked representation of the Quantity-like class
new_array = np.ma.array(data_masked_units)
assert_equal(data_masked.data, new_array.data)
assert_equal(data_masked.mask, new_array.mask)
# Test sharing the mask
data_masked.mask = [True, False, False]
assert_equal(data_masked.mask, new_array.mask)
assert_(new_array.sharedmask)
# Get the masked representation of the Quantity-like class
new_array = np.ma.array(data_masked_units, copy=True)
assert_equal(data_masked.data, new_array.data)
assert_equal(data_masked.mask, new_array.mask)
# Test that the mask is not shared when copy=True
data_masked.mask = [True, False, True]
assert_equal([True, False, False], new_array.mask)
assert_(not new_array.sharedmask)
# Get the masked representation of the Quantity-like class
new_array = np.ma.array(data_masked_units, keep_mask=False)
assert_equal(data_masked.data, new_array.data)
# The change did not affect the original mask
assert_equal(data_masked.mask, [True, False, True])
# Test that the mask is False and not shared when keep_mask=False
assert_(not new_array.mask)
assert_(not new_array.sharedmask)
class TestClassWrapping:
# Test suite for classes that wrap MaskedArrays
def setup_method(self):
m = np.ma.masked_array([1, 3, 5], mask=[False, True, False])
wm = WrappedArray(m)
self.data = (m, wm)
def test_masked_unary_operations(self):
# Tests masked_unary_operation
(m, wm) = self.data
with np.errstate(divide='ignore'):
assert_(isinstance(np.log(wm), WrappedArray))
def test_masked_binary_operations(self):
# Tests masked_binary_operation
(m, wm) = self.data
# Result should be a WrappedArray
assert_(isinstance(np.add(wm, wm), WrappedArray))
assert_(isinstance(np.add(m, wm), WrappedArray))
assert_(isinstance(np.add(wm, m), WrappedArray))
# add and '+' should call the same ufunc
assert_equal(np.add(m, wm), m + wm)
assert_(isinstance(np.hypot(m, wm), WrappedArray))
assert_(isinstance(np.hypot(wm, m), WrappedArray))
# Test domained binary operations
assert_(isinstance(np.divide(wm, m), WrappedArray))
assert_(isinstance(np.divide(m, wm), WrappedArray))
assert_equal(np.divide(wm, m) * m, np.divide(m, m) * wm)
# Test broadcasting
m2 = np.stack([m, m])
assert_(isinstance(np.divide(wm, m2), WrappedArray))
assert_(isinstance(np.divide(m2, wm), WrappedArray))
assert_equal(np.divide(m2, wm), np.divide(wm, m2))
def test_mixins_have_slots(self):
mixin = NDArrayOperatorsMixin()
# Should raise an error
assert_raises(AttributeError, mixin.__setattr__, "not_a_real_attr", 1)
m = np.ma.masked_array([1, 3, 5], mask=[False, True, False])
wm = WrappedArray(m)
assert_raises(AttributeError, wm.__setattr__, "not_an_attr", 2)

View File

@ -0,0 +1,294 @@
"""Miscellaneous functions for testing masked arrays and subclasses
:author: Pierre Gerard-Marchant
:contact: pierregm_at_uga_dot_edu
"""
import operator
import numpy as np
import numpy._core.umath as umath
import numpy.testing
from numpy import ndarray
from numpy.testing import ( # noqa: F401
assert_,
assert_allclose,
assert_array_almost_equal_nulp,
assert_raises,
build_err_msg,
)
from .core import filled, getmask, mask_or, masked, masked_array, nomask
__all__masked = [
'almost', 'approx', 'assert_almost_equal', 'assert_array_almost_equal',
'assert_array_approx_equal', 'assert_array_compare',
'assert_array_equal', 'assert_array_less', 'assert_close',
'assert_equal', 'assert_equal_records', 'assert_mask_equal',
'assert_not_equal', 'fail_if_array_equal',
]
# Include some normal test functions to avoid breaking other projects who
# have mistakenly included them from this file. SciPy is one. That is
# unfortunate, as some of these functions are not intended to work with
# masked arrays. But there was no way to tell before.
from unittest import TestCase # noqa: F401
__some__from_testing = [
'TestCase', 'assert_', 'assert_allclose', 'assert_array_almost_equal_nulp',
'assert_raises'
]
__all__ = __all__masked + __some__from_testing # noqa: PLE0605
def approx(a, b, fill_value=True, rtol=1e-5, atol=1e-8):
"""
Returns true if all components of a and b are equal to given tolerances.
If fill_value is True, masked values considered equal. Otherwise,
masked values are considered unequal. The relative error rtol should
be positive and << 1.0 The absolute error atol comes into play for
those elements of b that are very small or zero; it says how small a
must be also.
"""
m = mask_or(getmask(a), getmask(b))
d1 = filled(a)
d2 = filled(b)
if d1.dtype.char == "O" or d2.dtype.char == "O":
return np.equal(d1, d2).ravel()
x = filled(
masked_array(d1, copy=False, mask=m), fill_value
).astype(np.float64)
y = filled(masked_array(d2, copy=False, mask=m), 1).astype(np.float64)
d = np.less_equal(umath.absolute(x - y), atol + rtol * umath.absolute(y))
return d.ravel()
def almost(a, b, decimal=6, fill_value=True):
"""
Returns True if a and b are equal up to decimal places.
If fill_value is True, masked values considered equal. Otherwise,
masked values are considered unequal.
"""
m = mask_or(getmask(a), getmask(b))
d1 = filled(a)
d2 = filled(b)
if d1.dtype.char == "O" or d2.dtype.char == "O":
return np.equal(d1, d2).ravel()
x = filled(
masked_array(d1, copy=False, mask=m), fill_value
).astype(np.float64)
y = filled(masked_array(d2, copy=False, mask=m), 1).astype(np.float64)
d = np.around(np.abs(x - y), decimal) <= 10.0 ** (-decimal)
return d.ravel()
def _assert_equal_on_sequences(actual, desired, err_msg=''):
"""
Asserts the equality of two non-array sequences.
"""
assert_equal(len(actual), len(desired), err_msg)
for k in range(len(desired)):
assert_equal(actual[k], desired[k], f'item={k!r}\n{err_msg}')
def assert_equal_records(a, b):
"""
Asserts that two records are equal.
Pretty crude for now.
"""
assert_equal(a.dtype, b.dtype)
for f in a.dtype.names:
(af, bf) = (operator.getitem(a, f), operator.getitem(b, f))
if not (af is masked) and not (bf is masked):
assert_equal(operator.getitem(a, f), operator.getitem(b, f))
def assert_equal(actual, desired, err_msg=''):
"""
Asserts that two items are equal.
"""
# Case #1: dictionary .....
if isinstance(desired, dict):
if not isinstance(actual, dict):
raise AssertionError(repr(type(actual)))
assert_equal(len(actual), len(desired), err_msg)
for k, i in desired.items():
if k not in actual:
raise AssertionError(f"{k} not in {actual}")
assert_equal(actual[k], desired[k], f'key={k!r}\n{err_msg}')
return
# Case #2: lists .....
if isinstance(desired, (list, tuple)) and isinstance(actual, (list, tuple)):
return _assert_equal_on_sequences(actual, desired, err_msg='')
if not (isinstance(actual, ndarray) or isinstance(desired, ndarray)):
msg = build_err_msg([actual, desired], err_msg,)
if not desired == actual:
raise AssertionError(msg)
return
# Case #4. arrays or equivalent
if ((actual is masked) and not (desired is masked)) or \
((desired is masked) and not (actual is masked)):
msg = build_err_msg([actual, desired],
err_msg, header='', names=('x', 'y'))
raise ValueError(msg)
actual = np.asanyarray(actual)
desired = np.asanyarray(desired)
(actual_dtype, desired_dtype) = (actual.dtype, desired.dtype)
if actual_dtype.char == "S" and desired_dtype.char == "S":
return _assert_equal_on_sequences(actual.tolist(),
desired.tolist(),
err_msg='')
return assert_array_equal(actual, desired, err_msg)
def fail_if_equal(actual, desired, err_msg='',):
"""
Raises an assertion error if two items are equal.
"""
if isinstance(desired, dict):
if not isinstance(actual, dict):
raise AssertionError(repr(type(actual)))
fail_if_equal(len(actual), len(desired), err_msg)
for k, i in desired.items():
if k not in actual:
raise AssertionError(repr(k))
fail_if_equal(actual[k], desired[k], f'key={k!r}\n{err_msg}')
return
if isinstance(desired, (list, tuple)) and isinstance(actual, (list, tuple)):
fail_if_equal(len(actual), len(desired), err_msg)
for k in range(len(desired)):
fail_if_equal(actual[k], desired[k], f'item={k!r}\n{err_msg}')
return
if isinstance(actual, np.ndarray) or isinstance(desired, np.ndarray):
return fail_if_array_equal(actual, desired, err_msg)
msg = build_err_msg([actual, desired], err_msg)
if not desired != actual:
raise AssertionError(msg)
assert_not_equal = fail_if_equal
def assert_almost_equal(actual, desired, decimal=7, err_msg='', verbose=True):
"""
Asserts that two items are almost equal.
The test is equivalent to abs(desired-actual) < 0.5 * 10**(-decimal).
"""
if isinstance(actual, np.ndarray) or isinstance(desired, np.ndarray):
return assert_array_almost_equal(actual, desired, decimal=decimal,
err_msg=err_msg, verbose=verbose)
msg = build_err_msg([actual, desired],
err_msg=err_msg, verbose=verbose)
if not round(abs(desired - actual), decimal) == 0:
raise AssertionError(msg)
assert_close = assert_almost_equal
def assert_array_compare(comparison, x, y, err_msg='', verbose=True, header='',
fill_value=True):
"""
Asserts that comparison between two masked arrays is satisfied.
The comparison is elementwise.
"""
# Allocate a common mask and refill
m = mask_or(getmask(x), getmask(y))
x = masked_array(x, copy=False, mask=m, keep_mask=False, subok=False)
y = masked_array(y, copy=False, mask=m, keep_mask=False, subok=False)
if ((x is masked) and not (y is masked)) or \
((y is masked) and not (x is masked)):
msg = build_err_msg([x, y], err_msg=err_msg, verbose=verbose,
header=header, names=('x', 'y'))
raise ValueError(msg)
# OK, now run the basic tests on filled versions
return np.testing.assert_array_compare(comparison,
x.filled(fill_value),
y.filled(fill_value),
err_msg=err_msg,
verbose=verbose, header=header)
def assert_array_equal(x, y, err_msg='', verbose=True):
"""
Checks the elementwise equality of two masked arrays.
"""
assert_array_compare(operator.__eq__, x, y,
err_msg=err_msg, verbose=verbose,
header='Arrays are not equal')
def fail_if_array_equal(x, y, err_msg='', verbose=True):
"""
Raises an assertion error if two masked arrays are not equal elementwise.
"""
def compare(x, y):
return (not np.all(approx(x, y)))
assert_array_compare(compare, x, y, err_msg=err_msg, verbose=verbose,
header='Arrays are not equal')
def assert_array_approx_equal(x, y, decimal=6, err_msg='', verbose=True):
"""
Checks the equality of two masked arrays, up to given number odecimals.
The equality is checked elementwise.
"""
def compare(x, y):
"Returns the result of the loose comparison between x and y)."
return approx(x, y, rtol=10. ** -decimal)
assert_array_compare(compare, x, y, err_msg=err_msg, verbose=verbose,
header='Arrays are not almost equal')
def assert_array_almost_equal(x, y, decimal=6, err_msg='', verbose=True):
"""
Checks the equality of two masked arrays, up to given number odecimals.
The equality is checked elementwise.
"""
def compare(x, y):
"Returns the result of the loose comparison between x and y)."
return almost(x, y, decimal)
assert_array_compare(compare, x, y, err_msg=err_msg, verbose=verbose,
header='Arrays are not almost equal')
def assert_array_less(x, y, err_msg='', verbose=True):
"""
Checks that x is smaller than y elementwise.
"""
assert_array_compare(operator.__lt__, x, y,
err_msg=err_msg, verbose=verbose,
header='Arrays are not less-ordered')
def assert_mask_equal(m1, m2, err_msg=''):
"""
Asserts the equality of two masks.
"""
if m1 is nomask:
assert_(m2 is nomask)
if m2 is nomask:
assert_(m1 is nomask)
assert_array_equal(m1, m2, err_msg=err_msg)