1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
|
# PicoStream
High-performance data acquisition from PicoScope 5000a series to HDF5, with decoupled live visualisation. Designed for robust, high-speed logging where data integrity is critical.
## Quick Start
Install with `uv`:
```bash
uv pip install -e .
```
Install the [PicoSDK](https://www.picotech.com/downloads) (Linux users: see [Arch AUR wiki](https://aur.archlinux.org/packages/picoscope) for configuration).
Launch the GUI:
```bash
just gui
# or: uv run python -m picostream.main
```
Or use the CLI:
```bash
uv run picostream -s 62.5 -o data.hdf5 --plot
```
## Key Features
- **Producer-consumer architecture** with large shared memory buffer pool prevents data loss
- **Decoupled live plotting** reads from HDF5 file, cannot interfere with acquisition
- **Live-only mode** for continuous monitoring without filling disk
- **Numba-accelerated decimation** for efficient visualisation of large datasets
- **GUI and CLI interfaces** for interactive and scripted workflows
## Usage
### GUI Application
```bash
just gui
```
Configure acquisition parameters, start/stop capture, and view live data. Settings persist between sessions.
### Building Standalone Executable
```bash
just build-gui
```
Executable appears in `dist/PicoStream` (or `dist/PicoStream.exe` on Windows).
### Command-Line Interface
Standard acquisition (saves all data):
```bash
uv run picostream -s 62.5 -o my_data.hdf5 --plot
```
Live-only mode (limits file size):
```bash
uv run picostream -s 62.5 --plot --max-buff-sec 60
```
View existing file:
```bash
uv run python -m picostream.dfplot /path/to/data.hdf5
```
Run `uv run picostream --help` for all options.
## Documentation
### Architecture
PicoStream uses a producer-consumer pattern to ensure data integrity:
- **Producer (`PicoDevice`)**: Interfaces with PicoScope hardware, streams ADC data into shared memory buffers in a dedicated thread
- **Consumer (`Consumer`)**: Retrieves filled buffers from queue, writes to HDF5, returns empty buffers to pool in separate thread
- **Buffer Pool**: Large shared memory buffers (100+ MB) prevent data loss if disk I/O slows
- **Live Plotter (`HDF5LivePlotter`)**: Reads from HDF5 file on disk, completely decoupled from acquisition
This ensures the critical acquisition path is never blocked by disk I/O or GUI rendering.
### Data Analysis with `PicoStreamReader`
The `PicoStreamReader` class provides efficient access to HDF5 files with on-the-fly decimation.
```python
import numpy as np
from picostream.reader import PicoStreamReader
with PicoStreamReader('my_data.hdf5') as reader:
sample_rate_sps = 1e9 / reader.sample_interval_ns
print(f"File contains {reader.num_samples:,} samples at {sample_rate_sps / 1e6:.2f} MS/s")
# Iterate through file with 10x decimation
for times, voltages_mv in reader.get_block_iter(
chunk_size=10_000_000, decimation_factor=10, decimation_mode='min_max'
):
print(f"Processed {voltages_mv.size} decimated points")
```
### API Reference: `PicoStreamReader`
#### Initialisation
**`__init__(self, hdf5_path: str)`**
Initialises the reader. File is opened when used as context manager.
#### Metadata Attributes
Available after opening:
- `num_samples: int` - Total raw samples in dataset
- `sample_interval_ns: float` - Time interval between samples (nanoseconds)
- `voltage_range_v: float` - Configured voltage range (e.g., `20.0` for ±20V)
- `max_adc_val: int` - Maximum ADC count value (e.g., 32767)
- `analog_offset_v: float` - Configured analogue offset (Volts)
- `downsample_mode: str` - Hardware downsampling mode (`'average'` or `'aggregate'`)
- `hardware_downsample_ratio: int` - Hardware downsampling ratio
#### Methods
**`get_block_iter(self, chunk_size: int = 1_000_000, decimation_factor: int = 1, decimation_mode: str = "mean") -> Generator`**
Generator yielding `(times, voltages)` tuples for entire dataset. Recommended for large files.
- `chunk_size`: Number of raw samples per chunk
- `decimation_factor`: Decimation factor
- `decimation_mode`: `'mean'` or `'min_max'`
**`get_next_block(self, chunk_size: int, decimation_factor: int = 1, decimation_mode: str = "mean") -> Tuple | None`**
Retrieves next sequential block. Returns `None` at end of file. Use `reset()` to restart.
**`get_block(self, size: int, start: int = 0, decimation_factor: int = 1, decimation_mode: str = "mean") -> Tuple`**
Retrieves specific block from file.
- `size`: Number of raw samples
- `start`: Starting sample index
**`reset(self) -> None`**
Resets internal counter for `get_next_block()`.
### API Reference: `HDF5LivePlotter`
PyQt5 widget for real-time visualisation of HDF5 files. Can be used standalone or embedded in custom applications.
#### Initialisation
**`__init__(self, hdf5_path: str = "/tmp/data.hdf5", update_interval_ms: int = 50, display_window_seconds: float = 0.5, decimation_factor: int = 150, max_display_points: int = 4000)`**
- `hdf5_path`: Path to HDF5 file
- `update_interval_ms`: Update frequency (milliseconds)
- `display_window_seconds`: Duration of data to display (seconds)
- `decimation_factor`: Decimation factor for display
- `max_display_points`: Maximum points to display (prevents GUI slowdown)
#### Methods
**`set_hdf5_path(self, hdf5_path: str) -> None`**
Updates monitored HDF5 file path.
**`start_updates(self) -> None`**
Begins periodic plot updates.
**`stop_updates(self) -> None`**
Stops periodic updates.
**`save_screenshot(self) -> None`**
Saves PNG screenshot. Called automatically when 'S' key is pressed.
#### Usage Example
```python
from PyQt5.QtWidgets import QApplication
from picostream.dfplot import HDF5LivePlotter
app = QApplication([])
plotter = HDF5LivePlotter(
hdf5_path="my_data.hdf5",
display_window_seconds=1.0,
decimation_factor=100
)
plotter.show()
plotter.start_updates()
app.exec_()
```
## Changelog
### Version 0.2.0
- Added live-only mode with `--max-buff-sec` option
- Added GUI application for interactive control
- Improved plotter to handle buffer resets gracefully
- Added total sample count tracking across buffer resets
- Skip verification step in live-only mode for better performance
### Version 0.1.0
- Initial release with core streaming and plotting functionality
## Acknowledgements
This package began as a fork of [JoshHarris2108/pico_streaming](https://github.com/JoshHarris2108/pico_streaming) (unlicensed). Thanks to Josh for the original architecture.
|
