[helgew]

I feel like y’all are going in circles here. Emporía clearly has a business model that does not include providing a local API. Complaining and speculating about this is pretty futile, in my opinion, but obviously everyone is free to decide how they spend their time.

I will no longer be monitoring this thread, so if anyone needs help with the emporia downloader tool, please contact me through GitHub.

[helgew]

I believe the certificate thing may occur during the initial setup.

That is not how SSL works at all. If the client uses a client certificate, this will need to be pre-installed and is used to authenticate the client to the server during every new connection made by the client. Conversely, the server certificate is requested by/sent to the client on every new connection as well to verify that the server is indeed the correct endpoint to connect to. Only after both server and client have established that the other is who they are claiming to be (the TLS handshake phase), they will negotiate the encryption protocol to use and exchange the necessary information.

Using a letsencrypt-generated certificate with the domain name of the internal MQTT server results in a failed TLS handshake with the VUE. Using a self-signed certificate with the domain name the VUE is trying to connect to results in an “unknown CA” (CA = certificate authority) error.

• This reply was modified 3 years, 4 months ago by helgew.

[helgew]

Have you actually tried this? A good SSL client most definitely would care who it actually is connecting to, otherwise man-in-the-middle attacks would be much easier to pull off.

Also, you are assuming that the VUE doesn’t use a client certificate to connect to the server, which would have to be signed by the server’s private key. Do you have any evidence that that isn’t the case?

[helgew]

The certificate needs to be created by you on your MQTT broker server. You can build one using Lets Encrypt.

I don’t think this is as easy as described, because the CN of the certificate would have to be the same as the VUE is trying to connect to.

[helgew]

I think both Jertel and @helgew use pyemvue but may make their own posts to get the data

My downloader predates pyemvue and is written in java. It does not use pyemvue at all, which is written in python.

The API already includes parameters to enable someone to download only a subset of data and if you run my downloader continuously as intended, it will be “gentle” by default, i.e. request a chunk of data every 5 minutes and only request data for those past 5 minutes. Of course, someone could potentially run it differently and re-download all historical data every second, but that is rather difficult to prevent on either the client side or the server side.

• This reply was modified 3 years, 5 months ago by helgew.

[helgew]

port 8883 is not enough to make it MQTT

Technically true, but a) Occam’s razor, and b) I was communicating with Emporia staff directly early last year when I first wrote my own API client and they did confirm that they had discussed firmware modifications to allow for publication to other MQTT servers. I do not know if the current firmware allows for over-the-air updates, but that would obviously be a requirement.

it’s all the topic stuff that makes it difficult to manage.  if you don’t have a server fast enough to pull the data from the topic before the next datapoint hits, you lose.  It’s double the overherd vs a single TCP server with no benefits.

I think there are some misconceptions about the MQTT protocol in the above. Messages to the same topic do not overwrite older messages unless they are both flagged as to be retained. Messages pertaining to a topic that has subscriptions using a persistent session are retained by the broker until they are retrieved. Quality-of-Service flags can ensure that messages are received on the server before they are discarded on the client. MQTT has very little overhead compared to other “standard” protocols like HTTPS. Obviously, one could design and implement a task-specific and scalable protocol with a lower overhead than MQTT, but why re-invent the wheel?

[helgew]

I doubt it’s actually mqtt as well, mqtt doens’t handle streams so well, but handles state very well.

It’s definitely mqtt, which is perfectly suited for this application! The data packets are fairly small (one number per channel and a channel id sent every second) and the protocol designed around unreliable networks situations often encountered in remote sensing applications.

Here is a bit of my device’s network traffic. 10.0.3.50 is the IP of my device and 8883 is the secure MQTT server port it is talking to:

09:20:50.623508 IP 10.0.3.50.61360 > ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883: Flags [P.], seq 443:886, ack 1, win 5596, length 443
09:20:50.696145 IP ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883 > 10.0.3.50.61360: Flags [.], ack 886, win 65535, length 0
09:20:52.624809 IP 10.0.3.50.61360 > ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883: Flags [P.], seq 886:1329, ack 1, win 5596, length 443
09:20:52.702914 IP ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883 > 10.0.3.50.61360: Flags [.], ack 1329, win 65535, length 0
09:20:54.641881 IP 10.0.3.50.61360 > ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883: Flags [P.], seq 1329:1772, ack 1, win 5596, length 443
09:20:54.713688 IP ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883 > 10.0.3.50.61360: Flags [.], ack 1772, win 65535, length 0
09:20:56.642053 IP 10.0.3.50.61360 > ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883: Flags [P.], seq 1772:2215, ack 1, win 5596, length 443
09:20:56.715170 IP ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883 > 10.0.3.50.61360: Flags [.], ack 2215, win 65535, length 0
09:20:58.630211 IP 10.0.3.50.61360 > ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883: Flags [P.], seq 2215:2658, ack 1, win 5596, length 443
09:20:58.703876 IP ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883 > 10.0.3.50.61360: Flags [.], ack 2658, win 65535, length 0
09:21:00.626028 IP 10.0.3.50.61360 > ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883: Flags [P.], seq 2658:3101, ack 1, win 5596, length 443
09:21:00.699799 IP ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883 > 10.0.3.50.61360: Flags [.], ack 3101, win 65535, length 0

[helgew]

The Emporia device works differently from the Sitesage devices! It uses the mqtt protocol, as I mentioned above and does not wait for any server-side connections to retrieve data (i.e. it works only in push mode, not in pull mode). MQTT’s encryption mechanism, unfortunately, is more robust than most HTTPS implementations and not easily susceptible to man-in-the-middle attacks that could be used for a transparent proxy-type approach, for example.

Without the ability to configure the mqtt server to talk to, the Emporia servers (running on Amazon’s cloud platform) will always be needed. For some folks, that has advantages as those servers retain data much longer than the device itself could. For those of us interested in real-time monitoring, they aren’t really needed.

[helgew]

The device does not have to be a server of any kind. Being able to configure it’s mqtt client to push data to a local mqtt server though is well within its capability. For my purposes, that would be sufficient.

[helgew]

Good suggestion, @kpnobvious! I have generated a first public release in the hopes that it may be helpful. I will work on adding a simple UI and generating Windows and Mac executables as well.

[helgew]

Since this topic is getting a bit more attention lately, I have put in a little bit more work into my own java-based API client implementation. Apart from a few bug fixes, the downloader now supports periodic runs and all Emporiá supported time periods for which data can be downloaded. Since the API access parameters have been published by others, they are now pre-configured in the code and it is basically ready to run.

Hopefully, the updated documentation makes it straightforward to use, but please let me know if you have any questions or suggestions.

[helgew]

FWIW – there are several approaches to using the app’s API posted above, including my own. I think people overestimate the market share of HA enthusiasts that would be drawn to a more public API. I know, there are dozens of us, but in reality, there are hundreds for everyone of us who are just as happy to just use the app.

As for turning off or intercepting the flow to the emporía servers, that’s unfortunately near impossible without overwriting the firmware in the ESP (getting access to its pins may be the least of the problems going that route). The devices use encrypted MQTT messages, which, unlike HTTPS traffic, can not easily be accessed via MITM approaches, for example (I tried… I can do transparent HTTPS proxies with decryption, but not MQTT… maybe someone else is smarter there than I am).

[helgew]

@dekes1: if you have Java installed on your PC, you should be able to just create a file with all the properties configured, name it ‘config.properties’ and place it in the same folder as the emporia-download jar file. Double-clicking the file will start a background process that downloads your Vue data and either saves them to a JSON file (set the property ‘raw’ to the filename) or puts them into an Influx database (this is probably the best way).

I will work on creating a real Windows executable with a simple GUI that allows for the configuration of the necessary parameters.

• This reply was modified 4 years, 7 months ago by helgew.
• This reply was modified 4 years, 7 months ago by helgew.

[helgew]

I posted a link to my little project above, but I wanted to expand a little on what you can do with my Emporiá Vue Data Downloader: Once installed , the downloader will continuously pull data for all your devices at a customizable frequency and resolution just like the app would. It can also download historical data with the same limitation as the CSV download, obviously. The data can be saved directly to an Influx database for use by your home assistant tools or it can be output in JSON format to a file or to a pipe for your own processing, for example publishing to your own MQTT server.

Right now, the downloader is run from the command line but works on all platforms. I can provide easier ways to configure and run the tool if there is enough interest. I am also open to suggestions for other data formats to output.

@marty-emporia: I had been emailing about this with Ted and Franz, but have not received a reply as to whether you all are OK with making this project more publicly available. Please let me know.

[helgew]

I am also very interested in the option to use my own MQTT server! In the meantime and in the absence of an official API, this project might be of interest to some!

[Author]

Posts