Building a weather system with Jan Goolsbey

Welcome to the CircuitPython Show. I'm your host, Paul Cutler. This episode I welcome

back Jan Goolsbey, also known in the Adafruit community as cgrover. Jan first visited the

show back in episode 33, which I've linked in the show notes. Jan, welcome back to the

show.

Hello, Paul. How are you today?

I'm doing great. Thanks for making time and coming back on. I really appreciate it.

Well, I enjoy the discussions with you and I'm looking forward to this one.

I am too, so you've been working on a weather station for a number of years.

Tell me how it all got started.

Well, it got started many years ago when I, when we moved into this house, we have this

detached garage that I call my workshop.

It does have an area set aside for that.

And I brought a collection of my electronics projects.

And back then I wasn't doing a lot of software, it was mostly hardware.

So I had, you know, an IBM 5150 personal computer.

I had Intel 8080 parts and resistors and capacitors, you know, the whole thing.

And I was throwing them out in this detached garage where we have in the

workshop that was unheated.

And I noticed after about a year, components started to decay and especially

the metal leads, like on integrated circuits would get this corrosion layer on

them and it made it really difficult to solder.

They didn't work in the sockets anymore.

And so I was getting a little bit discouraged.

I also noticed some damage to my metallic tools, you know, things like wrenches and screwdrivers.

We're starting to show some corrosion on the outside of that.

And I never thought that would happen because we live in the desert here in Washington State.

You just don't expect that to happen.

But we have wet winters.

You know, even though we get less than six inches of rain in a year, it happens in the

winter and that's when things are colder and that's when condensate forms.

Well, you get the drift there. I was needing to upgrade the garage so that I could store things out there and work on things out there without worrying about corrosion. So I get this new obsession. The new obsession was, how do I solve this corrosion? So I did some reading about it. And the first thing I needed to do was measure temperature and humidity. And that was, that was really where it all starts. And, and if you can determine when condensate forms,

then you can probably prevent it. And that's really what I wanted to do. I didn't want to put a heater out in the garage right away. I wanted to figure this thing out first. So that's where this started. And fortunately, about that same time, MicroPython came onto the scene. And Tony DeCola was working on putting MicroPython on some Adafruit M0 boards. And I thought, well, this is perfect. Okay, you know, I had done some Arduino programming. But Arduino doesn't, you know, it's not as good as Arduino. And I thought, well, this is perfect. Okay, you know, I had done some

doesn't work with my brain. I'm a hardware guy and so I had to do compiling and all this kind of stuff.

Yeah, I know how to do that, but that's not the fun part of it for me. So I wanted to get something

like MicroPython where I could very quickly put together a prototype, start to characterize what

the temperature and humidity issues were, and see if I can solve this corrosion problem. And Tony did

some great work with getting MicroPython introduced in some Adafruit boards. And I, at the same time,

Lady Ada came up with a board with an RF transmitter on it. It was the RFM69, an 800MHz transceiver that could send and receive little tiny data packets, you know, 10 to 15 character data packets. Nothing fancy, nothing fast. And I thought, but that's perfect because I just need temperature and humidity. And since it's a detached garage, I didn't want to have to go out there and read the display all the time. So I made a little unit that was, you know, a little bit more compact. And I thought, well, I'm going to have to make it a little bit more compact. And I thought, well, I'm going to have to make it a little bit more

an M0 with the RFM69 built into it, put micropython on it, stuck it out in the garage,

transmitted it to a tiny little remote that I made with an OLED display, and all I did was I just

watched temperature and humidity just to try to figure out when does corrosion occur. Meanwhile

I'm doing some studying because when you get that data, well what's the first thing you do,

at least for me, throw it in a spreadsheet. Start looking at things like dew point and the capacity

air to hold moisture and all the kind of stuff that leads to corrosion. I started to refine the

software to convert the humidity and the temperature into something that looked more like an index that

was related to whether I was experiencing a corrosion event. I was hoping to get to the

point where I could predict when a corrosion event occurred. That took me to the next phase of the project.

next phase of the project was setting aside that RFM69 and M0 and go to more of an internet solution

where I put a Pi portal out in the garage and with the same temperature humidity sensor

transmitted the data up to Adafruit I/O and got data trends and you know you've got a great

browser-based dashboard tool that I could use to look at the data and compare and get some feeling

for an even deeper feeling into how to solve this corrosion issue. And by about that time,

I figured, well, I better remodel this garage. It was built from surplus government lumber.

The whole garage was. It was very functional and it was retro-fantastic. It was great. As a matter

of fact, a friend of mine came out and filmed a short video in there about time travel because

they needed a workshop that a crazy scientist would use, you know, to build this time machine.

So they filmed the time machine portion in my workshop. And the next week was when I had the

crews come in and tear down the garage and put up a new one and build on the same footprint,

build the garage. And PiPortal was perfect timing for that. And I also put in a heat pump out in the

garage because what I had determined by then, I was using CircuitPython with the PiPortal,

I determined that even though there are some complex equations for figuring out how much moisture there is in the air and you know how it forms and and there are some standards for when you can expect certain kinds of materials to corrode, it really boiled down to keeping the garage above 32 degrees. And that way you avoid the condensation that obviously occurs when things freeze. And as long as you keep it at a comfortable temperature, somewhere above 32 degrees, you can pretty much avoid corrosion.

issues. And so if we finished the remodeling, we got the pipe portal out there, I'm watching the

data, and we had this great snow storm and I, you know, backed the car back into the garage next to

where the workshop is, and about 15 minutes later the corrosion alarm goes off and I, oh, I brought

moisture into the garage that wasn't natural. What happened? And the snow melted, we get this

corrosion event, and that created a whole new set of concerns for me that how do now how do I get

rid of the moisture in the garage. And thanks to CircuitPython, I can modify the PyPortal monitor

to give me the data that I need to look at different algorithms for solving these problems.

And prototypically, almost immediately, you can study the data, change the algorithms,

and completely change the functionality of the monitor to suit your whole new set of theories

about how you're going to predict and prevent these things. Well, since I was in the problem-solving

mode there and needed to get rid of the moisture in the garage, that's when I determined there's

a really tight link between internal corrosion and external corrosion. The data and the conditions

that can cause internal corrosion are driven by the exterior weather conditions, as well

as the heat pump and all those things. It got me thinking about how do I incorporate

external weather conditions

with the internal workshop conditions and get a better feel for

how I can prevent and how I can be even this case mitigate the

collection of the moisture in the workshop

That's when I started going down this this path of taking a look at open weather

map.org because they have a free weather condition service for local conditions and

And that worked pretty well. I had one device that looked at the external weather and one device that looked at the corrosion monitor stuff.

And I, you know, went between the two and started to try to make a correlation between those.

It became rather difficult to do that. I had spreadsheets and I couldn't correlate the data very well because I couldn't collect it in one place.

That's when I had the idea of uploading the weather condition data that I was getting from openweathermap.org into AIO, so I had streams that would show up on my browser dashboard.

And then I could start mapping one temperature set against the other and make a determination of what the relationship is between inside and outside conditions.

So I knew when to open the garage door, turn on the fan, and blow the moisture back out

into the environment.

Which doesn't work, by the way, when it's raining, because you can't fit any more moisture

in the outside air, which makes sense.

Good to know.

Yeah, I'll have to write up this whole corrosion monitoring obsession one of these days.

I'll let you know when that comes about.

I'm having too much fun programming what I'm working on right now.

And what I'm working on right now is this evolution of that into a combined weather

system that looks at internal sensors, specialty sensors that I might have locally, and weather

conditions.

And it was kind of all spawned by this AIO collection of the data feeds and the way that

I could look at that.

So I worked on an architecture where I can upload that data, where I can bring it down

to devices.

And I got to the point where I could take all that data and put it into spreadsheets

or make comparisons, but I had the collection of the data from the two sources.

And that whole thing was working fine.

And I was learning more about how I'm going to do this corrosion monitoring, how I'm going

to set these alarms.

And there's still work to be done.

know, a wrench got thrown into the works and that was openweathermap.org decided that they were going

to start charging for their service and I can't blame them. It's a great service. But the model

that they used was, at least at the time when I was making the consideration, I don't know if it's

changed, they required a credit card to be able to get data from them even though they called it a

free service. And you know I'm just not trusting enough to give somebody my credit card say here's

here's a blank check for you, you know?

And I thought, well, I'll just look for some other ones.

And I ran across weather.gov, which is a NOAA service,

the National Oceanic and Atmospheric Administration there.

I knew I could figure that one.

NOAA had this really great service

that had this fantastic detail of these weather stations

that were really close to my home.

And I thought, this is two miles away at an airport

that's real close by here.

They had a weather station.

I thought, this is perfect.

It's not just some consolidated weather data that I might get from open weather

map, which they do a good job, but they average weather stations in the area

to come up with their, their number.

I thought this is going to work great.

Well, what I found out was these little tiny weather stations, they

have peppered all over the place.

If you just look at one of them, it's not very reliable.

So, and I thought, okay, well, that's fine.

I'll just go, I'll do what openweathermap.org does.

I'll look at 20 different weather stations and I'll average the data.

And you can imagine I'm taxing CircuitPython pretty significantly.

I'm trying to do this on a Pi portal with an M4 processor, even though it has a

separate ESP32 processor on there, the M4 just doesn't have enough memory to do all

that internet, you know, downloading all the JSON files that are necessary, decoding

that.

And then of course I kind of went nuts and I needed a fancy display and that eats up

memory too. The poor little pipe portal was just suffering. Pretty soon I found that that would be an impossible task. And I really didn't know where to go from there. I didn't want to have to go back to openweathermap.org, although that would have been a viable solution. I was pouring through some Adafruit IO documentation because I'm trying to solve some of the reliability issues I was having using the M4 and memory capacity issues. And I found out that there's an Apple

weather kit plus up in AIO plus that I hadn't realized was there.

And I had been using AIO plus because, you know, I have lots of AIO feeds,

because I have lots of data sources.

So I was already paying for this, but I just didn't realize that this plus up to

the Apple weather kit was out there and it gives an open weather map kind of view

of the world, including five-day forecasts, you know, some data that I wasn't using that could be very useful. This is a no-brainer. We're just going to go with that. And that's where I started to think about this collection of data that I have as an architecture. Since I was facing that choice of determining where the source was going to be for this weather data, I got about a dozen Pi portals, I needed to make it compatible with that. That's when I started to develop that architecture. And that's where I started to think about how I could use that data to make my architecture more compatible with the weather data. And that's where I started to think about how I could use that data to make

and version one of that architecture is in an article, it's documented in an article out in the

Adafruit Playground. That was something that the objective of that architecture was to get that

the two weather conditions, you know, the internal and external weather condition data into a single

place, but it was also to try to preserve the pie cordless that I used for displays.

Since they were limited, I had to try to find an architecture that could support the need for the

PyPortal to have a bunch of data, but keep it from having to go out to the internet and get these

gigantic data files, JSON files, and decode them itself. It just couldn't do that. And so the first

architecture was designed to take care of that. So what were the challenges with the first

architecture of the weather station? Well let me recap just for a second and go

back to the you know the zero architecture was just a point-to-point

thing with that two-way radio kind of a setup and then you know I saw the

obvious need at that point to use AIO so the architecture became more AIO centric

and then I started gravitating to the Pi portal and that's that's kind of the

Lexus of the issue and when you work with the Pi portal you have a lot of overhead contained in

the drivers that make the Pi portal easy to use and easy to program but if you're trying to expand

and take in a bunch of feeds and I had a lot of AIO feeds of course you start running out of memory

and with that M4 processor and so I got to the point where I saw the need to do something different

and yet I still tried making the M4 work. You know, I bumped up against the memory all the time,

and I got rid of some of the fancy graphics and put in some more pedestrian stuff, and I just

wasn't very happy with it. So I came up with this idea that I needed to modify the architecture so

that I could still look at weather data. And the other goal was I wanted to continue to use the M4s,

the Pi portals, but be more sensitive to the memory issues and some other issues that they have,

so that I could still continue to use my investment in Pi portals, because I have many of them.

But I needed to move to something, and I'm very reluctant to move to something new, because

I had it working, I thought, and but also I'm cheap and I'd like to use the Pi portals wherever

possible. I struggle with the notion that once you write some software, you make it as good as you

possibly can and you shouldn't ever have to change it, which that's a myth. And so I get

trapped into that often. I try to talk myself into a philosophy that I learned a while ago

when I was making printed circuit boards is, yeah, it's okay to start from scratch. It's

okay to start over. You probably learned something since the last time you coded that and it's

okay to recode it and you'll probably use something better the next time around. So

limitations in the Pi portal predominantly. I wanted to have multiple display devices.

And the first architecture was kind of funny, the way I tried to synchronize all these asynchronous

devices. If you have four or five displays and they're all accessing AIO, you run into

the throttle limits of the Adafruit I/O. And if you have a free subscription to AIO, you're

limited to 30 transactions per minute, which doesn't sound like a lot, and it isn't when

you have multiple devices. Fortunately, I was using AIO Plus and I had the limit of

60 transactions per minute, and double that other rate. And I certainly appreciate the

fact that they're trying to give this service to a whole bunch of people and they need to

throttle it that way. So I would go to each one of my devices and I'd say, "Okay, the

to AIO at 10 minutes past the hour, get your business done.

And then the weather monitor can go out to AIO

and extract weather information

at 15 or 20 minutes past the hour.

And the second weather display, the one that, or the, yeah,

the second weather display that I have is in the living room

is a matrix portal.

And I said, well, you can go at 20 or 30 minutes

past the hour and get the data.

And you can only go out there every 20 minutes

get the data. I had this really complicated formula trying to synchronize all these devices

that were, that I wanted to be autonomous and it just couldn't work. Because if you

ever had an internet error, if you know what those are, which we have a few, the wireless

connection would break or something and then the device would have to resynchronize and

it would heterodyne with the other devices and you'd still end up with throttle failures.

and I just didn't know how to fix that. Well, I think Brent and Tyath, who do the development

work for Adafruit in the AIO realm, they came up with a really simple scheme for providing

throttling information. And I thought, well, I've tried everything else. Let's try this

and see if we can make it work. And I hesitated to try their new scheme for throttling because

I figured it would be way more complicated than what I could do. I'm not an expert programmer.

I ran into so many troubles trying to synchronize anyway that this throttling stuff is going

to be a scheme that will take me a while to learn how to do. I was wrong. It was so simple.

That's a good problem to have.

It was surprisingly simple. I should have known better. I should have checked it out

and played with it. Well, anyway, I know better now. But Tyep did most of the work on that

And it's just brilliant what he did just brilliant and I put in you know one or two extra lines of code

Whenever I had to go out and do something with AIO

And all it did was it did a query and it says how much throttled do I have left?

And that's all controlled them by the server rather than by my devices and my devices was just wait until there's enough throttle left to

Do something so Adafruit IO actually has part of the API that you can query it to say how many throttling requests?

Do I have left in this minute?

Yes, and and surprisingly that going out and querying how much throttle you have left doesn't seem to affect your throttle limit

Which just you know, that's a real plus

So whenever I have a transaction with AIO now, I go out and check that throttle. Well that forced me to go to a new architect

So I have an architecture version 2 that I put in place and this is the version 2 is called remix

and it's out in the Playground. If you're interested in it, there are two articles that I wrote for Playground. One is the structure of the first architecture, and the second is the structure of the second architecture. And it also gives the code examples.

And I'll make sure I link to those in the show notes as well.

And I welcome as many comments as I can get on that, because I'm hoping that I'm using a throttling correctly. It seems to work.

Because now with the second version of the architecture, I've tested with six devices and four of those are Pi portals. One is an ESP32 S3 repeater is what I call it.

but it takes the place of the corrosion monitor out in the garage and it picks up the Apple WeatherKit information, translates that and moves it back into AIO feeds that I can then access with PyPortals.

And I'm only getting that small chunk of data that I need rather than the whole JSON file.

So I can get the pi portals to continue to work because I limit how much data they get when

they're accessing AIO. They don't have to get the whole thing. They only get what they need.

So I've had six devices running with the new throttling stuff and it's just been flawless.

The only errors I have much more related to, you know, my need for fancy graphics and things like

that where I ran out of memory and that, you know, that's a personal issue that I'll just

have to resolve somehow. Well tell me a little bit about the interface because I've seen pictures

and I think most of the listeners would be interested in knowing more about it. Right now

there are four different user interfaces that I use. The corrosion monitor, even before it came

into this new architecture, was an Elkar's Star Trek style interface. It isn't touch sensitive or

or anything, but it still uses that feature. And it's built very similarly to the displays

that they actually use. Not necessarily in the code or the graphics design, although

that does kind of mirror it, but it uses a layered graphics approach. So the background

image has all of your indicators in it. Like, I'm using the internet now, or I'm going out

I'm looking at my thermometer or I'm using the SD drive to store information for that wonderful spreadsheet that I always keep track and

You cover those up with masks instead of drawing the object like an antenna or

Wireless access that's in the background and you just cover it up with a colored square. So the interface really is kind of graphical

reminiscent of the L cars and it uses some trickery to make that work which is a little stage magic which I think is kind

Of a nod to how they did things on Star Trek, too

So that's the corrosion monitor and that that has been modified somewhat and is coming to the new environment

the weather monitor was one based on something that John Park did quite a while ago on the Pi portal and

I've adapted it and changed the graphics a little bit

We call it Mikey because our local weather man. Its first name is Mike

And I recorded a snippet of his voice introducing the fact that we've turned the weather monitor on

So every time we turn it on we get our favorite forecaster telling us that he's gonna give us the web. That is pretty neat

the other interface is the matrix portal weather station and I used an m4 initially, but

There again, I started running in performance problems. So I switched that one to

the Matrix Portal S3. So it's a fully capable ESP32-S3. Lots of memory available for goofing

around with graphics and doing things. Technically I could download entire JSON files and things like

that, but I still kept it in kind of a simple display and it shows an abbreviated version of

the weather and then in the scrolling bar down below it tells us what the wind is gusting to,

it tells us what the shop temperature is and it scrolls by and we find that one probably to be

the most useful out of those interfaces the first three interfaces and the fourth interface is one

that I'm not happy with well it kind of looks like a raspberry pi booting up it's just text

and that's the translator that sits out in the garage and I really need it to be more of a clock

and I needed it to, well frankly, I needed it to look more like an LCARS interface. So,

the next version of that, which I'm working on right now, and I do have a link and some pictures

in that second Playground article, it shows a more expansive LCARS interface and it combines

the external weather with the internal shop conditions in a single display. It can either

be used as that repeater that translates data and sends it back to AIO or it can be used as a

standalone display. That's the up and coming for it. Will there be a modification to the architecture?

Well, I don't sit still on this stuff so probably a version 3 but right now I need to perfect that

new Elkars interface and that's running on a let's see that's running with a 3.4 inch

TFT feather wing with an ESP32S3 feather plugged into the backup.

So it it's kind of like a Pi Portal which I dearly love but it's it's not yet a Pi Portal.

I hope in the future to add some other bells and whistles to it so it has sound because we've got

to get Mikey to talk. And also I like to when I put these in an enclosure I like to have a

inside the enclosure in case here in the desert things get hot, you know, and you've got to turn

on the fan inside the enclosure. And that's, those are things that I need to add. Is there anything

else that comes to mind for what's next in the project? Of course, I'm still on that journey

to figure out how do I adequately predict corrosion? What weather changes outside and

conditions inside, including, you know, back in the car and that's, that has snow all over and

and under it that's gonna melt.

How do I predict when that corrosion event's going to occur?

And there's some science in that.

And right now I focus on dew point

because dew point seems to be, that's a magic number

'cause that tells you when the air is saturated with water,

that the temperature at which the air gets saturated

with water is the dew point.

Well, that tells you a lot about corrosion

because if the air is saturated with water

and you have something metallic that's a little bit colder,

it's going to condense water on that right away.

And if you compare internal dew point to external dew point,

that tells you a little bit about the other condition

I talked about where if I want to take the water

in the workshop and push it outside with a fan,

is there room outside to accept that moisture?

And dew point can tell you some of that.

It's not a true measure of capacity of the air outside,

but it's a pretty good indicator, I think.

But what I struggle with is really gaining an understanding

about how much moisture do I have in the workshop

and how much moisture can the outside tolerate?

How much can I give it?

So that's the next step is to do a little science

and try to figure this thing out.

And then you would think that would be enough.

the natural extension is now how am I going to use the features of AIO to control the

internet connected heat pump that's in the garage so that I can have it automatically

abate it while I'm sitting on a beach somewhere. Sure. Well a good project is never over. No,

I'll find something. Well that's fascinating. I appreciate you coming on the show. I appreciate

you letting me talk about this because I really like the feedback that I get. And there's

I need to add to this besides thanking you for the opportunity to have these discussions that I really enjoy. I would like to thank, of course, Tyath and Brent for the work that they've done on AIO to make it so easy to use and for adding all these features in AIO plus like, you know, Apple Weather Kit that I'm using now. But I also like to go back to like the first architecture that I put in place with the two-way radio kind of a setup. And I think that's really, really cool. And I think that's really, really cool. And I think that's really, really cool. And I think that's

up, that was made possible by a lot of work that Tony DiCola did, because he introduced

MicroPython into that environment very creatively and made that really useful. So I appreciate

all the work that he put into that and the leadership that he had shown back then to

see the possibility to put MicroPython on these little devices. And of course, the evolution

the CircuitPython has just made them even easier to use. And then Jerry N, Jerry Neal,

he was a developer for Apple back in those days, and I think he's a part-time developer now. Jerry N, his handle on Discord, he did some really breakthrough work in getting the two-way radio drivers for CircuitPython, and those were game-changers for me that got me into this whole rattle that I'm traveling through.

through now and having fun with the corrosion bond.

- That's great.

- So I wanna thank those people.

You know, the community has really been supportive

of all these things and giving me lots of suggestions

and ideas and so I appreciate that.

- I'll make sure I share the learn guides in the show notes

and hopefully you'll get some more feedback from it.

- That's super, it's something I look forward to.

- Jan, thanks so much for being on the show.

- Oh, you're welcome and thanks for having me.

- Thanks to Jan for coming on the show

his weather station, its architecture, and how CircuitPython and Adafruit I/O+ have helped

his project.

And remember what Jan said - if you have any feedback after listening or looking at the

project on the Adafruit Playground, let him know!

For show notes, transcripts, and more, visit www.circuitpythonshow.com.

Until next time, stay positive.

(electronic music)