diff options
| author | Sam Scholten | 2025-10-15 10:35:24 +1000 |
|---|---|---|
| committer | Sam Scholten | 2025-10-15 11:04:02 +1000 |
| commit | 55001da9a7add8ae57c66f7c921e4f70309df511 (patch) | |
| tree | 9109491dc9f2a5a58660d76b37473d866a145c72 | |
| parent | 14042a8f9e36f2bf61d23f60dfba223eafb58157 (diff) | |
| download | picostream-55001da9a7add8ae57c66f7c921e4f70309df511.tar.gz picostream-55001da9a7add8ae57c66f7c921e4f70309df511.zip | |
docs: improve README
| -rw-r--r-- | README.md | 231 |
1 files changed, 150 insertions, 81 deletions
@@ -1,147 +1,216 @@ # PicoStream -PicoStream is a high-performance Python application for streaming data from a PicoScope to an HDF5 file, with an optional, decoupled live visualization. It is designed for robust, high-speed data logging where data integrity is critical. +High-performance data acquisition from PicoScope 5000a series to HDF5, with decoupled live visualisation. Designed for robust, high-speed logging where data integrity is critical. -## Features +## Quick Start -- **Robust Architecture**: Separates data acquisition from disk I/O using a producer-consumer pattern with a large, shared memory buffer pool to prevent data loss. -- **Zero-Risk Live Plotting**: The plotter reads from the HDF5 file, not the live data stream. This ensures that a slow or crashing GUI cannot interfere with data acquisition. -- **Efficient Visualization**: Uses `pyqtgraph` and a Numba-accelerated min-max decimation algorithm to display large datasets with minimal CPU impact. -- **Live-Only Mode**: Optional buffer size limiting for pure real-time monitoring without generating large files. -- **Flexible Data Reading**: The included `PicoStreamReader` class allows for easy post-processing, including on-the-fly decimation ('mean' or 'min_max'). +Install with `uv`: -## Installation +```bash +uv pip install -e . +``` -1. `pip install -e .` +Install the [PicoSDK](https://www.picotech.com/downloads) (Linux users: see [Arch AUR wiki](https://aur.archlinux.org/packages/picoscope) for configuration). -2. Install the official PicoSDK from Pico Technology. Linux can take some additional work, see the AUR wiki for details. +Launch the GUI: -## Usage +```bash +just gui +# or: uv run python -m picostream.main +``` + +Or use the CLI: -The primary way to use the package is through the `picostream` command-line tool. +```bash +uv run picostream -s 62.5 -o data.hdf5 --plot +``` + +## Key Features -### Acquisition +- **Producer-consumer architecture** with large shared memory buffer pool prevents data loss +- **Decoupled live plotting** reads from HDF5 file, cannot interfere with acquisition +- **Live-only mode** for continuous monitoring without filling disk +- **Numba-accelerated decimation** for efficient visualisation of large datasets +- **GUI and CLI interfaces** for interactive and scripted workflows -The following commands show different acquisition modes: +## Usage + +### GUI Application ```bash -# Standard acquisition (saves all data) -picostream -s 62.5 -o my_data.hdf5 --plot +just gui +``` -# Live-only mode (limited buffer) -picostream -s 62.5 --plot --max-buff-sec 60 +Configure acquisition parameters, start/stop capture, and view live data. Settings persist between sessions. + +### Building Standalone Executable + +```bash +just build-gui ``` -Run `picostream --help` for a full list of options. +Executable appears in `dist/PicoStream` (or `dist/PicoStream.exe` on Windows). -### Live-Only Mode +### Command-Line Interface -For pure real-time monitoring without saving large amounts of data, use the `--max-buff-sec` option: +Standard acquisition (saves all data): ```bash -picostream -s 62.5 --plot --max-buff-sec 60 +uv run picostream -s 62.5 -o my_data.hdf5 --plot ``` -This limits the HDF5 file to the last 60 seconds of data, automatically overwriting older data. Perfect for continuous monitoring applications where you only need recent data visible. +Live-only mode (limits file size): -**Benefits:** -- Prevents disk space issues during long acquisitions -- Maintains full plotting functionality -- File size stays constant after initial buffer fill -- No performance impact on acquisition - -### Viewing an Existing File +```bash +uv run picostream -s 62.5 --plot --max-buff-sec 60 +``` -The plotter can be run as a standalone tool to view any compatible HDF5 file. +View existing file: ```bash -python -m picostream.dfplot /path/to/your/data.hdf5 +uv run python -m picostream.dfplot /path/to/data.hdf5 ``` -## Data Analysis with `PicoStreamReader` +Run `uv run picostream --help` for all options. -The output HDF5 file contains raw ADC counts and metadata. The `PicoStreamReader` class in `picostream.reader` is the recommended way to read and process this data. +## Documentation -### Example: Processing a File in Chunks +### Architecture -Here is an example of how to use `PicoStreamReader` to iterate through a large file and perform analysis without loading the entire dataset into memory. +PicoStream uses a producer-consumer pattern to ensure data integrity: + +- **Producer (`PicoDevice`)**: Interfaces with PicoScope hardware, streams ADC data into shared memory buffers in a dedicated thread +- **Consumer (`Consumer`)**: Retrieves filled buffers from queue, writes to HDF5, returns empty buffers to pool in separate thread +- **Buffer Pool**: Large shared memory buffers (100+ MB) prevent data loss if disk I/O slows +- **Live Plotter (`HDF5LivePlotter`)**: Reads from HDF5 file on disk, completely decoupled from acquisition + +This ensures the critical acquisition path is never blocked by disk I/O or GUI rendering. + +### Data Analysis with `PicoStreamReader` + +The `PicoStreamReader` class provides efficient access to HDF5 files with on-the-fly decimation. ```python import numpy as np from picostream.reader import PicoStreamReader -# Use the reader as a context manager with PicoStreamReader('my_data.hdf5') as reader: - # Metadata is available as attributes after opening sample_rate_sps = 1e9 / reader.sample_interval_ns - print(f"File contains {reader.num_samples:,} samples.") - print(f"Sample rate: {sample_rate_sps / 1e6:.2f} MS/s") + print(f"File contains {reader.num_samples:,} samples at {sample_rate_sps / 1e6:.2f} MS/s") - # Example: Iterate through the file with 10x decimation - print("\nProcessing data with 10x 'min_max' decimation...") + # Iterate through file with 10x decimation for times, voltages_mv in reader.get_block_iter( chunk_size=10_000_000, decimation_factor=10, decimation_mode='min_max' ): - # The 'times' and 'voltages_mv' arrays are now decimated. - # Process the smaller data chunk here. - print(f" - Processed a chunk of {voltages_mv.size} decimated points.") - - print("\nFinished processing.") + print(f"Processed {voltages_mv.size} decimated points") ``` -## API Reference: `PicoStreamReader` +### API Reference: `PicoStreamReader` + +#### Initialisation -The `PicoStreamReader` provides a simple and efficient interface for accessing data. +**`__init__(self, hdf5_path: str)`** -### Initialization +Initialises the reader. File is opened when used as context manager. -#### `__init__(self, hdf5_path: str)` -Initializes the reader. The file is opened and metadata is read when the object is used as a context manager. +#### Metadata Attributes -### Metadata Attributes +Available after opening: -These attributes are populated from the HDF5 file's metadata when the reader is opened. +- `num_samples: int` - Total raw samples in dataset +- `sample_interval_ns: float` - Time interval between samples (nanoseconds) +- `voltage_range_v: float` - Configured voltage range (e.g., `20.0` for ±20V) +- `max_adc_val: int` - Maximum ADC count value (e.g., 32767) +- `analog_offset_v: float` - Configured analogue offset (Volts) +- `downsample_mode: str` - Hardware downsampling mode (`'average'` or `'aggregate'`) +- `hardware_downsample_ratio: int` - Hardware downsampling ratio -- `num_samples: int`: Total number of raw samples in the dataset. -- `sample_interval_ns: float`: The time interval between samples in nanoseconds. -- `voltage_range_v: float`: The configured voltage range (e.g., `20.0` for ±20V). -- `max_adc_val: int`: The maximum ADC count value (e.g., 32767). -- `analog_offset_v: float`: The configured analog offset in Volts. -- `downsample_mode: str`: The hardware downsampling mode used during acquisition (`'average'` or `'aggregate'`). -- `hardware_downsample_ratio: int`: The hardware downsampling ratio used. +#### Methods -### Data Access Methods +**`get_block_iter(self, chunk_size: int = 1_000_000, decimation_factor: int = 1, decimation_mode: str = "mean") -> Generator`** -#### `get_block_iter(self, chunk_size: int = 1_000_000, decimation_factor: int = 1, decimation_mode: str = "mean") -> Generator` -Returns a generator that yields data blocks as `(times, voltages)` tuples for the entire dataset. This is the recommended method for processing large files. -- `chunk_size`: The number of *raw* samples to read from the file for each chunk. -- `decimation_factor`: The factor by which to decimate the data. -- `decimation_mode`: The decimation method (`'mean'` or `'min_max'`). +Generator yielding `(times, voltages)` tuples for entire dataset. Recommended for large files. -#### `get_next_block(self, chunk_size: int, decimation_factor: int = 1, decimation_mode: str = "mean") -> Tuple | None` -Retrieves the next sequential block of data. Returns `None` when the end of the file is reached. Use `reset()` to start over. +- `chunk_size`: Number of raw samples per chunk +- `decimation_factor`: Decimation factor +- `decimation_mode`: `'mean'` or `'min_max'` -#### `get_block(self, size: int, start: int = 0, decimation_factor: int = 1, decimation_mode: str = "mean") -> Tuple` -Retrieves a specific block of data from the file. -- `size`: The number of *raw* samples to retrieve. -- `start`: The starting sample index. +**`get_next_block(self, chunk_size: int, decimation_factor: int = 1, decimation_mode: str = "mean") -> Tuple | None`** -#### `reset(self) -> None` -Resets the internal counter for `get_next_block()` to the beginning of the file. +Retrieves next sequential block. Returns `None` at end of file. Use `reset()` to restart. +**`get_block(self, size: int, start: int = 0, decimation_factor: int = 1, decimation_mode: str = "mean") -> Tuple`** + +Retrieves specific block from file. + +- `size`: Number of raw samples +- `start`: Starting sample index + +**`reset(self) -> None`** + +Resets internal counter for `get_next_block()`. + +### API Reference: `HDF5LivePlotter` + +PyQt5 widget for real-time visualisation of HDF5 files. Can be used standalone or embedded in custom applications. + +#### Initialisation + +**`__init__(self, hdf5_path: str = "/tmp/data.hdf5", update_interval_ms: int = 50, display_window_seconds: float = 0.5, decimation_factor: int = 150, max_display_points: int = 4000)`** + +- `hdf5_path`: Path to HDF5 file +- `update_interval_ms`: Update frequency (milliseconds) +- `display_window_seconds`: Duration of data to display (seconds) +- `decimation_factor`: Decimation factor for display +- `max_display_points`: Maximum points to display (prevents GUI slowdown) + +#### Methods + +**`set_hdf5_path(self, hdf5_path: str) -> None`** + +Updates monitored HDF5 file path. + +**`start_updates(self) -> None`** + +Begins periodic plot updates. + +**`stop_updates(self) -> None`** + +Stops periodic updates. + +**`save_screenshot(self) -> None`** + +Saves PNG screenshot. Called automatically when 'S' key is pressed. + +#### Usage Example + +```python +from PyQt5.QtWidgets import QApplication +from picostream.dfplot import HDF5LivePlotter + +app = QApplication([]) +plotter = HDF5LivePlotter( + hdf5_path="my_data.hdf5", + display_window_seconds=1.0, + decimation_factor=100 +) +plotter.show() +plotter.start_updates() +app.exec_() +``` ## Changelog ### Version 0.2.0 -- **New Feature**: Added live-only mode with `--max-buff-sec` option for buffer size limiting -- **Enhancement**: Improved plotter to handle buffer resets gracefully -- **Enhancement**: Added total sample count tracking across buffer resets -- **Enhancement**: Skip verification step in live-only mode for better performance -- **Documentation**: Updated README with live-only mode usage examples +- Added live-only mode with `--max-buff-sec` option +- Added GUI application for interactive control +- Improved plotter to handle buffer resets gracefully +- Added total sample count tracking across buffer resets +- Skip verification step in live-only mode for better performance ### Version 0.1.0 - Initial release with core streaming and plotting functionality ## Acknowledgements -This package began as a fork of [JoshHarris2108/pico_streaming](https://github.com/JoshHarris2108/pico_streaming) (which is unlicensed). I acknowledge and appreciate Josh's original idea/architecture. +This package began as a fork of [JoshHarris2108/pico_streaming](https://github.com/JoshHarris2108/pico_streaming) (unlicensed). Thanks to Josh for the original architecture. |
