Chapter 2 - Filtering data

In the previous chapter, you learned how to read and print data that is a bit raw. Now, try to select a few columns and handle them properly.

Start with these two columns: Time (time) and Magnitude (mag). After getting the information from these columns, filter and adapt the data. Try formatting the date to Qt types.

There is not much to do for the Magnitude column, as it’s just a floating point number. You could take special care to check if the data is correct. This could be done by filtering the data that follows the condition, “magnitude > 0”, to avoid faulty data or unexpected behavior.

The Date column provides data in UTC format (for example, 2018-12-11T21:14:44.682Z), so you could easily map it to a QDateTime object defining the structure of the string. Additionally, you can adapt the time based on the timezone you are in, using QTimeZone.

The following script filters and formats the CSV data as described earlier:

 1
 2import argparse
 3import pandas as pd
 4
 5from PySide2.QtCore import QDateTime, QTimeZone
 6
 7
 8def transform_date(utc, timezone=None):
 9    utc_fmt = "yyyy-MM-ddTHH:mm:ss.zzzZ"
10    new_date = QDateTime().fromString(utc, utc_fmt)
11    if timezone:
12        new_date.setTimeZone(timezone)
13    return new_date
14
15
16def read_data(fname):
17    # Read the CSV content
18    df = pd.read_csv(fname)
19
20    # Remove wrong magnitudes
21    df = df.drop(df[df.mag < 0].index)
22    magnitudes = df["mag"]
23
24    # My local timezone
25    timezone = QTimeZone(b"Europe/Berlin")
26
27    # Get timestamp transformed to our timezone
28    times = df["time"].apply(lambda x: transform_date(x, timezone))
29
30    return times, magnitudes
31
32
33if __name__ == "__main__":
34    options = argparse.ArgumentParser()
35    options.add_argument("-f", "--file", type=str, required=True)
36    args = options.parse_args()
37    data = read_data(args.file)
38    print(data)
39

Now that you have a tuple of QDateTime and float data, try improving the output further. That’s what you’ll learn in the following chapters.