With the USB scope hookup from yesterday’s post it’s also quite easy to capture data for further experiments. It helps to have a fixed data set while comparing algorithms, so I used the DSO-2090 software again to capture over a million 8-bit samples.
The dump is binary data, but decoding the format is trivial. Some header bytes and then each value as 2-byte int. Just to show that JeeMon is also up to tasks like this, here’s the “rec2text.tcl” script I created for it:
And this is how it was used to generate a text file for the plots below:
(silly me – I’m still impressed when I see a script process over a million items in the blink of an eye…)
That’s a bit too much data for quick tests, but here are the first 10,000 values from the 75W-bulb-on-20-VAC:
There’s quite a bit of noise in there if you look more closely:
The measurements are all over the map, with values over almost the entire 0..255 range of the scope’s 8-bit ADC. The actual spikes are probably even larger, occasionally.
I hope that this is a decent dataset to test out filtering and DSP techniques. It’ll be much quicker to try things out on my Mac than on a lowly ATmega 8-bit MPU, and since both can run the same C & C++ code, it should be easy to bring things back over to the ATmega once it it does the right thing.
Hmmm… now where do I find suitable DSP filtering algorithms and coefficients?
Update – Found a great resource, thx Paul & Andreas. Stay tuned…
There is software for filtering old audio recordings from vinyl (or even shellac) discs and tape-recordings. It is supposed to be able to reduce noise and clicks/pops – might just be the right tool for this job?
Given that you’re really only after the 50/60Hz signal, wouldn’t a little tuned analogue circuitry of the LC style be helpful?
Could do. I’d like to have brief go at digital filtering, though. It’ll be a quick test, promise ;)
I have successfully used http://www-users.cs.york.ac.uk/~fisher/mkfilter/ to generate C code for digital filtering. As an example of how efficient it can be, take a look at the graphs at http://weber.fi.eu.org/blog/Bricolage/Arduino/tyco_controller_adc.html . I used there a 4th order Butterworth filter with a -3dB cutoff @ 5Hz.
Maybe a bit off-topic, but there are several ICs available with are specially made for power metering, for example the Maxim 78M6613. While using a standard IC is a lot less fun, it probably is more accurate and safer ;)
Thanks for the info – while I’m aware of those, I’m not after measurements – just ON / OFF detection and low cost.
JC, this is a nice idea to take these USB scope data samples to develop and improve algorithms on the PC. Just take into consideration that the sample rate and input bandwidth of your USB scope is likely to be totally different to your AVRs on the Jeenode. The AVR does not normally ‘see’ all this noise that you see in the USB scope data. At least not with the required basic RC filtering on the ADC input which is absolutely necessary for other reasons.
BR, Jörg.
how about that:
http://interactive-matter.eu/2009/12/filtering-sensor-data-with-a-kalman-filter/
Thanks for the suggestion and the useful link. I’ll try a few things in the next few days. Probably will want to stick to integer processing, but I’ll see if I can compare Kalman as well.
I’m holding my breath to read your next post! Had the same problem with voltage and current while building my open energy monitors http://openenergymonitor.org/ …
I am not as technically savvy as you are, so I’m crossing the fingers so you find a solution, I’ll surely learn something from it :)
Good luck!
Digital filtering should always be the absolutely last step in processing an analog signal after all the things done ‘in front’ of it are done correctly. If you have artefacts in the data from wrong signal conditioning, no digital filter is able to remove them. Consider sampling frequency components above the nyquist frequency. These will produce ‘mixing’ products, which are mirrored into the measured frequency range. This mixing products cannot be removed by any (low pass) filter. So the first thing to do is to remove frequencies above half the sample frequency as best as possible in the analog world (e.g. RC low pass filter). Then, a simple sliding average is one of the best filters available with regard to efficiency (performance/’cost’) in this sort of mearurments. Just compare the results of sliding average to your ‘complicated’ filters and decide if its worth the effort.
BR, Jörg.
(Psst, I’ll confirm your sliding average remarks in a few days…)
Haven’t thought about the analog filtering. I (now) understand the requirement to filter below Niquist freq regardless of how clever the digital filter is made. I’m hoping the following will suffice: a 10/10 kΩ divider (a bit high, but less current draw that way), with a cap (1µ?) over lower R. Then feed it to the analog pin with a 1k resistor, with a (0.1µ) cap from analog to ground, which creates a low-pass. Sampling at 10 KHz.
:-)
some words about moving average filtering: http://www.dspguide.com/ch15.htm
This is from one of the best books I have ever read about digital signal processing. And very affordable, too :-)
BR, Jörg.
Great – that’s the one I’ve been reading.
The best filter I have seen on a atmel is a home brew of moving average, and moving standard deviation which was used to remove outliers. For my case over 95% were beyond three deviations from the average so that was what I used for the cutoff to remove the outliers. The moving average quickly and efficiently cleaned up the last bit. I think I kept 50 points to work on and recalculated the standard dev every third point. Sadly that code died with a hard drive about a year ago.
JC, your RC filtering is a bit too severe. With the 10K divider and a 1uF cap, this is already a 20db per decade filter with corner frequency about 32Hz. (The following RC combination filters further from about 1.6Khz, but in complicated way since the two filter sections interact) Just remove the 1uf and this makes the corner at ~250Hz with the attenuation greater than 24db at the Nyquist frequency
Thx Martyn – good, so it looks like the 1kΩ / 0.1µ on the ADC input does enough. But note that the 1µ cap is not used as filter but to generate a stable VCC / 2 voltage for the other end of the shunt. Does it still act as a filter in such a case? I really ought to draw the planned schematic again one of these days.
JC, this is exactly how I understood your explanation of the circuit. The 1u serves to stabilize the reference (‘middle’) voltage and that seems to be a good choice.
Let’s see about the results. I must admit that I liked your idea to take a bigger number of samples in the ‘real’ circuit and store to flash or eeprom and analyze later on. Seems to be a good way to better understand what is really going on.
Oops, sorry, I “drew” the 1uf the wrong side of the sampling resistor. If you leave it in, it acts as a “boost” to higher frequencies since it is parallel with the 5K source impedance of the Vcc/2 bias. Perhaps you could redraw for confirmation & I’ll recalculate (or just leave it off if you don’t want to increase the 1K & 0.1uF values)
BTW, if you intend to investigate later the VA consumption by multiplying by the sampled instantaneous V, the time lag introduced by the filter(s) needs compensating for.
Just use the same filter components for voltage measurements too, then there is no need for extra compensation.
Clever! Ok, so I could have 0.1 Ω shunt as one input and a 200 + 1 kΩ resistor divider as the other (using 2x 100 kΩ to reduce voltage drop over each R). Then use a 1 kΩ + 0.1 µF as RC for each leg to get rid of HF noise. Both signals will swing around 0, but the 1 kΩ limits current through the ESR diodes, and I can just measure the positive halves of each.
Hm, it gets more complicated if I’d want to also sense voltage and power the thing without transformer. In that case, one side of the shunt really has to end up at ground level for the circuit. That means the other side will drop below 0 half the time – probably harmless with a 1 kΩ resistor between shunt and ADC in. But it does mean that the signal becomes an awkward one to filter (all the negative halves clipped).
I’ll stick with the setup as outlined for now. Will draw a new schematic soon, once I get the whole kaboodle a bit more stable. Got some nice posts coming up for y’all :)