API to pull data

[jh26]

I want to automate a daily pull of my data so that I can run some automatic analysis. Would like a RESTful API instead of the just the manually activated CSV email.

[DrGuns]

Curious what analysis you are running. I’ve been trying to figure out how I could go about pulling the data for a couple of my expansion module channels Heat & A/C(need both since I have a heat pump system) aggregate and pull down local temperature data from something like OpenWeatherMap.org and plot the heating/cooling power usage relative to outside temp and what I know to be my inside temp(Plotting the differential essentially). Then I could compare what kind of effect changing the temperature setpoint of the thermostat gives me. Or what my before and after looks like on Insulating and air sealing rooms in my house(to see if I’m earning that money back). I’ve never messed with pulling down data through API’s. Hopefully one of these days I can get the free time and can learn how.

[Sfstowers]

I’m working on a way to download “degree-days” and try to correlate with energy usage. The industry seems to use this measure, my electric bill has the degree-days for the month on the bill. It looks like about 2/3 of our energy use is for the HVAC, so getting a better handle on this makes lots of sense.

[akifbayram]

Would love to be able to integrate this into Home Assistant to trigger certain actions based on power usage (washer/dryer finished, abnormal power usage, etc..)

[miguelr]

I would love to be able to import the data and use it on my own grafana dashboards. Something like InfluxDB would be amazing, or even some live raw data output from the device that I can access from a web browser (think ADS-B-like)

[eviloreo]

I would really like to see an API as well! Might also make getting a web interface easier if it uses the API 😀

• This reply was modified 4 years, 10 months ago by eviloreo. Reason: Edited after seeing the other request I had; WebUI

[saltcreep]

+1 API for HA integration

[sean_8395]

I’ll upvote this as a feature I’d like to see. If this was completed it could feed the other highly requested item of a web-based view.

[magillus]

I would love to see API access with live and history – almost same data that mobile App does.

 

[jc523]

Here’s another vote for an API.

I’m looking to integrate into home assistant.

[jbrukardt]

Just installed my vue yesterday, fantastic hardware and setup for the price.   Even the most basic of APIs would be extremely helpful.   Getting the data into influxDB and then onwards to homeassistant or grafana or numerous other things would allow for power alerts, better analytics, better cost savings calculations, etc.

 

A lot of this exists for iotawatt already due to its open nature, but the vue hardware is cheaper, and the system overall is more streamlined in my opinion

[charlie]

Same here. It would be great to integrate at least basic readings (including solar readings) with Hubitat and HA.

[jh26]

@drguns, if I had an API or even just the ability to pull the CSV file automatically instead of manually triggered, I would create a daily usage report emailed with usage and charts for each channel.  This would give me a daily health report on the well, water heater, pool, fridge, and solar system.

If I could do this, I would also buy a second Vue for my other house.

[Maddyn]

+1 for the API. looking at integrating with OpenHAB so I can have all my info in one location. Just any sort of access to the raw data would be better than nothing, then you could either parse, import, log or do whatever you want with it.

[almarcano]

I would like to say I will be very happy with an API to integrate with Home Assistant, thanks

[helgew]

I am also very interested in the option to use my own MQTT server! In the meantime and in the absence of an official API, this project might be of interest to some!

[dekes1]

Yes, please consider adding extensibility to the system via APIs!

Maybe as an even easier and quicker option for your development team would be to open up access to the raw data on your cloud server so we can build our own interfaces (like a local node that polls the cloud server data) so we can build integrations into SmartThings/Home Assistant/Hubitat.

[jh26]

Any sort of access to the raw data will do. It’s clear you have quite a number of technically talented customers. Open it up and we’ll do the work for you.

[Sfstowers]

I am new to Emporia, but I’ve been deeply involved in home automation for years.  I am overall very happy with the device but find data analysis very limited.  We have 400 amp service, so I bought 2 devices, one for each incoming main line pair.  The current app does not allow me to combine these 2 as a total usage, so it greatly limits my analysis.  I am writing my own program to accomplish this.  Direct access to data with an API call would be a huge step forward.  My experience has shown that HVAC usage is the prime determinate of our electricity consumption.  Without having concurrent temperature and humidity values, it’s impossible to critically assess and draw meaningful conclusions on our efforts at conservation.  I’m going to build an API interrogation routine in my program to the free api.weather.gov service to retrieve this data and mesh it with my electricity usage.  This would be a fairly simple service that you could build into your app that would greatly increase it’s impact.  Thanks!

[helgew]

I posted a link to my little project above, but I wanted to expand a little on what you can do with my Emporiá Vue Data Downloader: Once installed , the downloader will continuously pull data for all your devices at a customizable frequency and resolution just like the app would. It can also download historical data with the same limitation as the CSV download, obviously. The data can be saved directly to an Influx database for use by your home assistant tools or it can be output in JSON format to a file or to a pipe for your own processing, for example publishing to your own MQTT server.

Right now, the downloader is run from the command line but works on all platforms. I can provide easier ways to configure and run the tool if there is enough interest. I am also open to suggestions for other data formats to output.

@marty-emporia: I had been emailing about this with Ted and Franz, but have not received a reply as to whether you all are OK with making this project more publicly available. Please let me know.

[dekes1]

@helgew, very interested in the work you’ve done. Would/could you create a How-To to explain how to run the command tool from Windows?

[helgew]

@dekes1: if you have Java installed on your PC, you should be able to just create a file with all the properties configured, name it ‘config.properties’ and place it in the same folder as the emporia-download jar file. Double-clicking the file will start a background process that downloads your Vue data and either saves them to a JSON file (set the property ‘raw’ to the filename) or puts them into an Influx database (this is probably the best way).

I will work on creating a real Windows executable with a simple GUI that allows for the configuration of the necessary parameters.

• This reply was modified 4 years, 7 months ago by helgew.
• This reply was modified 4 years, 7 months ago by helgew.

[zerog2k]

another +1 for real options on direct access to our data:

1. also send the data to mqtt server (from local device preferred)

2. also send the data to influxdb server (from local device preferred)

3. pollable cloud api (less desirable for me, but still acceptable if it’s high-enough resolution, e.g. 5-10s)

Overall, I would think giving users option to additional have the device report to local mqtt and/or influxdb could be implemented comparatively quickly and easily, and probably cover 90% of the “home automation” power-users like myself and several others here. (Perhaps the trickiest part could be how to pass  configuration [securely] to device; I’m hand-waving a tad, but seems like much easier than the littany of concerns with architecture, cost, reliability, security, etc around exposing and maintaining a public-facing user api.)

 

• This reply was modified 4 years, 7 months ago by zerog2k.

[gercodries]

An API would be great, I won’t buy any products where I don’t have access to my own data in a convenient and automated fashion. I made that mistake with Ring and Nest and won’t make it again, no matter how “convenient” the product may appear to be. Separately from that, I really just want the hardware. No cloud services of any kind.

If I can disable the cloud integration, apps, etc. and just send the raw data to my MQTT server, I’m a happy camper: instant buy.

[MP-Ranger]

IFTTT would be ideal for this as it is possible to integrate different websites and collate the results onto a single spreadsheet like Google Docs.

[hsteinhauer]

I just purchased my unit. I am encouraged that there is work being done to pull the data to a local system.

I look forward to being involved with this project.

[smitters1501]

Hey guys, if you still waiting for an API, look at this open source project  to get the data of emporia vue…

https://pypi.org/project/pyemvue/

[magico13]

For those of you looking for home assistant integration, there’s always this project https://github.com/magico13/ha-emporia-vue and for those who want to put data into InfluxDB there’s this https://github.com/jertel/vuegraf

[almarcano]

You really are MAGICO… Thanks for helping our HA Community to grow! Well done…

[PGE-customer-SJC]

Thanks for this! If anyone has successfully managed to integrate this with PVOutput.org to import as consumption data that would be super.

[playfair]

I’m new to this product; just ordered my system yesterday! However, after reading many threads requesting API (starting over a year ago), then seeing in another post, “…our software, hardware, and business is developed around collection and analytics of energy data in the cloud”, I’m guessing that the energy usage data these devices send to emporia is being negotiated with outside companies? It would definitely explain the reluctance to make the Vue data privately accessible. I tried to read the “terms and conditions” to see the privacy statement in the app but the link was broken.

[PowEEr]

I agree… the low hardware price and nonSAAS model is probably a loss leader to sell usage data to 3rd parties.  I’d very much love an API that didn’t require cloud connectivity at all.  It probably wouldn’t take too much effort to man-in-the-middle the host/client session, although this is nominally against their ToS.  I’d much rather this unit talk to a Raspberry Pi versus a monetized cloud service.  Still, even a public API – that still required cloud connectivity – would be better than nothing.

[magico13]

I’m pretty sure the hardware is all ESP-32 based so someone could in theory write a custom firmware for it to all be local but it obviously wouldn’t work anymore with their app and cloud.

[PowEEr]

You wouldn’t need to rewrite the ESP32 FW… just man-in-the-middle.  I haven’t looked yet (I’ve only had the system installed for 24 hours), but I doubt the protocol protects against that vector.  Of course it is against their ToS.  Honestly rewriting the ESP-32 FW wouldn’t be too difficult… it’s obviously just a multiplexed ADC cycling through the sensors.  I haven’t looked to see which uP they’re using, but writing new FW wouldn’t take more than a week… if Emporia ever folds (e.g. after introducing a subscription model), I’ll happily take on that project.  As-is the software experience is just barely usable enough for me to not invest that effort.

[magico13]

Without pulling the private keys off the current firmware you’re not going to be able to MITM it, unless you’ve got a way to decrypt https traffic without the keys, so overwriting the current firmware should the cloud site go down is probably easier. Anyway this is getting off-topic enough that it should either be a different thread or otherwise discussed elsewhere.

[chrwei]

So here’s something to consider…

You have an app, that app talks to your servers via some authenticated protocol, and I’m assuming it’s using our user auth so that we can’t just guess a device id and get someone else’s data, right?  Is that not an API?

So, you already have an API and a user auth method, you just need to release some documentation for it.  Being able to integrate with HASS and OpenHAB and SmartThings and others would be a huge selling point for you. You don’t even have to document the whole API, just the auth and data fetch.

[helgew]

FWIW – there are several approaches to using the app’s API posted above, including my own. I think people overestimate the market share of HA enthusiasts that would be drawn to a more public API. I know, there are dozens of us, but in reality, there are hundreds for everyone of us who are just as happy to just use the app.

As for turning off or intercepting the flow to the emporía servers, that’s unfortunately near impossible without overwriting the firmware in the ESP (getting access to its pins may be the least of the problems going that route). The devices use encrypted MQTT messages, which, unlike HTTPS traffic, can not easily be accessed via MITM approaches, for example (I tried… I can do transparent HTTPS proxies with decryption, but not MQTT… maybe someone else is smarter there than I am).

[syzygy]

Hi,

I’d like to add to the interest here.  I’m a potential customer but not being able to integrate is a non-starter.  I love the effort people have put in so far and it shows the interest but this relies on an internal api that will break without warning.

Make an api and make it operable without cloud.  With a small new company I need to be sure I’ll be able to keep this working if you go under (my plum lightpads can attest to this, even my sunpower panels had bad data for many many months because they don’t know how to run a service)

Waiting to buy probably 2-3 of these when this is done.

[chrwei]

a local API for things like a smart switch is fairly easy, but for an energy monitor it is not.  Gen2 might have the resources to be able to provide the last value for each interval category, but that might sacrifice resources that could be used for other things, like longer offline caching, or a higher polling interval to make features like device type detection.

what would be easiest is to have it just listen on a TCP socket and anything that connects gets real time data streamed at it.  This would take very little code or device resources, and would allow the community to step in and provide services that Emporia doesn’t have the resources for.

[syzygy]

I’d be happy with any api.  Want to stream each sample?  Sure.  Making it local, publish it.  Api doesn’t have to be complex.

Then again the gen 1 was based on a dual core 240mhz chip so a lot should be possible.

I’d also rather an api than see anyone try something like device type detection.  Sense tried and failed spectacularly.

[RNMEnergy]

Another user here waiting for the API to be published. Writing just to let Emporía Know.

[mikemar87x]

+1 for API.

I managed to figure out how to put a copy of the export data via the emailed link (which used to be the same). So I just had a reminder on my phone to request a new report every week and then a PowerShell script i wrote would scrape the zip file and extract the CSV.

Sometime in December, the script stopped working and I think it’s because report generation changed. The name used to be “https://url/{customer_id}.zip”, so I could use consistently use curl to extract the file.

Now though, the mobile app lets you generate a custom report for a certain time period, which has a consequence of changing the filename of the outputted file from “{customer_id}.zip” to “{customer_id}-{report_id}.zip” where {report_id} = a random number generated for that specific report.

I am happy the reporting capability was improved, but it did break my process lol.

At this point, I’d rather not have to write a script to parse my mail inbox in order to get the link generated for the most recent report, and then have to remember to choose an interval for each report each week when I go to trigger it from the mobile app.

Could we get even just a simple API that allows us to specific an interval and time period and then spit out a report?

I did check out the open source third party utils for pulling data, but they’re a bit overkill for me. All I need to do is pull the data to a local JSON file for archival purposes. I don’t need to CRUD on devices themselves, just read the report data.

• This reply was modified 3 years, 9 months ago by mikemar87x.

[kpnobvious]

Hi,
I recently purchased my vue and got it installed.  Part of the reason i purchased it was the ability to make API calls.  There are already several projects mentioned in previous threads.  I use a modified version of Jertel’s to pull the data into an influxDB to graph with grafana.  I just use a raspberry pi to do it.  It was fairly straightforward to setup.

I know emporia’s API isn’t as well documented as say EcoBee or Flume but thanks to some of the folks here the ability to get the data is out there.  While that does require an internet connection, i really don’t think that’s an issue, nor do i see some sort of local API being implemented.

I can share some steps if people are interested.  I changed Jertel’s github project to run periodically vs continuously and worked out a simple dashboard in grafana.  I just need to work on some longer data views, like sum by day.

[bpendz]

@kpnobvious please do indulge me how you got this working on a Pi. I am unable to get this to run.

[kpnobvious]

Hi @Bpendz

Here’s a brief rundown:

1. PiZeroW with 32GB class 10 microSD
2. PiOS 5.4, fully updated
3. Install InfluxDB –
4. Install Grafana –
5. Install pip3 and supporting packages
apt install python3-pip, pip3 install pyemue, pip3 install influxdb, pip3 install influxdb-client
6. I git cloned PyEmVue to test , then git cloned vuegraf
I think i also did pip3 install -r requirements.txt from within each of the git folders to make sure i had all the requirements
7.  Once i got pyemvue to pull back my data, i setup the json file for vuegraf.
8. Connect grafana to the influxdb for vue.  Would probably be good to setup a reader account for grafana to the influxdb

Once everything was installed and running, i tested with pyemvue to make sure it would pull data.  I then got vuegraf working as released.

Here’s where I began changing a few things:
1. I had to modify the sample JSON to get pyemvue to tag properly, or else everything showed up as the channel name\# from Vue.  I can provide a sample if needed.
2. I modified a copy of pyemvue to change the logging from stdout to a log file and put in a for loop so it would run for 4.5 minutes before exiting.  I still have it pulling data in the same range.
3. I created a launcher script to call my modified pyemvue script and added it to crontab running every 5 minutes

I’ve spent some time messing with my grafana dashboard to suit my needs.  So far i like it.  I’m working on writing something using pyemvue to just pull usage by day as i havent’ figured out how to summarize the second data up by day and show a daily view in addition to the hourly views.

If i did it over again, i probably would have spent time figuring out dockers for everything with cloud backup.  The vuegraf database isn’t actually very large so far, but i’ve only had it about a week.

In the end i have this:

[bpendz]

Will try this later! Thank you

[waterboyz]

KPNobvious

I was trained on vacuum tubes and mainframes.  It is obvious I have not kept up based on your recent programming post.  But I am still interested.

 

 

• This reply was modified 3 years, 9 months ago by waterboyz.
• This reply was modified 3 years, 9 months ago by waterboyz.

[kpnobvious]

Don’t worry @waterboyz, I’m old enough to know what those things are!  I got into raspberry pis with Pi Hole for DNS filtering.  There are a lot of youtube videos out there that can you get started.

Really with all these projects we’re just impersonating our emporia app accounts to log in and pull data locally.  Potentially the things we do with the data could be added to the app.  It seems from this thread there are about a dozen people give or take using the data.  I’m very happy we have this ability at no additional charge.  One of the reasons i bought the vue over other products like miser or sense was individual circuit monitoring and these open source projects that let me pull my data locally.

If you or @Bpendz get stuck, i can start a new thread about my setup.

[helgew]

Since this topic is getting a bit more attention lately, I have put in a little bit more work into my own java-based API client implementation. Apart from a few bug fixes, the downloader now supports periodic runs and all Emporiá supported time periods for which data can be downloaded. Since the API access parameters have been published by others, they are now pre-configured in the code and it is basically ready to run.

Hopefully, the updated documentation makes it straightforward to use, but please let me know if you have any questions or suggestions.

[oziee]

+1 for MQTT

My whole home runs on HomeAssistant and every device functions offline. I don’t like the idea of relying on cloud based services for many reasons.

Would be just great to be able to consume the data myself with the unit feeding mqtt data.

Plus the data it shows does not provide accurate information that is used in Australia. Here we have different time based tariffs and solar feed in tariffs, so the current app is not sufficient for the viewing needs here. Having mqtt data I could create my own visual view of what actually is going on using HA

oz

[chrwei]

Not just Australia either!  in the “simple” midwest US we have tiered energy rates.  the $ figure Emporia provides is useless except where the grid is antique.  But that’s another post, and one exists already.

[kpnobvious]

Thanks @Helgew!  I’ll have to look at that again.  I think if there some steps to compile it on Windows, that might help out a lot of people, particularly some of those looking for daily ranges.  I presume you just need the JDK or even posted a compiled jar to github may help.

[helgew]

Good suggestion, @kpnobvious! I have generated a first public release in the hopes that it may be helpful. I will work on adding a simple UI and generating Windows and Mac executables as well.

[waterboyz]

Oziee

Please reach out to me about HAssistant.

Thanks…

Mike

northernmarietta

ATgmail.com

[jj613]

@helgew and @smitters1501

Where are you getting the documentation  for the Vue APIs that you are calling?

Could you describe the authentication steps more fully?   Are you using the same ID/password that you configured in the Vue phone app?  And then what are the steps to building the authentication headers, and what are all the available calls?  And I guess the most important question … how do you know?

[magico13]

@jj613 Short answer is by monitoring the network traffic of the app. See here for documentation, there might be other documentation out there as well that is more/less featured. https://github.com/magico13/PyEmVue/blob/master/api_docs.md

[KCole313]

Just another +1 for an API.

There must be some sort of API in place already. As others have pointed out, app is able to fetch data. Furthermore, I’m able to connect 3rd party services like Google and OhmConnect. If some documentation could just be released, that would be great!

[wz2b]

I think I want to make sure we are adding the word “local” to our API requests.  I don’t want to have to reach out to a cloud service for this.  I want rapid data, and I don’t necessarily even want it on the internet.

1. This device says it supports ZigBee SEP, which is a standard.  Does anybody know if you can communicate with it this way – set ZigBee network parameters, keys, that sort of thing?
2. Is there a local web interface, that might have a fetch API of some sort built into it?

 

A  number of embedded devices like this communicate to a server that can be redirected to a local “good enough” backend.  For those looking for a local API, maybe standing up a small server on an RPi like device, that emulates the “official” backend, plus a little DNS magic would make “all local” possible.

 

 

 

 

[wz2b]

The other thing that comes to mind is bluetooth.  The setup instructions talk about bluetooth; is the device BLE?  is live data available with GATT?

[John Polasek]

Replying to wz2b, 7155… Absolutely I’d like a totally local solution;  I bought the Emporia in order to replace a 12 year old Sitesage system that died and would have cost 6 times as much to replace.  The sitesage allowed me to query the local device for instantaneous usage from all channels as an xml stream, and my Indigo home control system has the ability to parse either XML or JSON http feeds into a “sensor device” with a list of channel values.  I’d do a once a minute query to keep track of HVAC and aerobic septic running minutes to know when to replace filters and add chlorine, as well as flashing an alert when the running washer and dryer stopped drawing power out in the garage.  I’m looking at how to use one of the github “unofficial” APIs as a bridge to pull this same data from the Emporia server, but worry about them closing the access down if the load gets too high on their system… if they would allow me to query the LOCAL device for that info, it wouldn’t load their central server at all.

[jj613]

@johnPolasek and others … I agree that an API would be useful, as would local access to the device by the app.  But let’s be realistic about how that might happen (if it ever does) and then if and how *local API access* could happen.   API access is easy … apparently it already exists, as some people have found and are already using so it’s just a matter of seeing where that evolves.

Clearly the device does not have an API server in it.  The device works by making API calls to the cloud-based servers.   So I suspect that if local access were ever to be provided it would likely be through some hokey rigged mechanism where the phone app connects, perhaps through Bluetooth, and provides some way, perhaps through the local Wifi network or directly through Bluetooth, for the device to report data to the app.   The phone app would be the “server”, or loosely, the recipient of API calls by the device.

I really doubt that the approach to provision  local would be by turning the device into an API server.  That’d be a big leap.  So combining those, local access *for an API* seems unlikely.  But not impossible and it would be great.

[helgew]

The device does not have to be a server of any kind. Being able to configure it’s mqtt client to push data to a local mqtt server though is well within its capability. For my purposes, that would be sufficient.

[John Polasek]

As I said, the cloud API that I am looking into will be sufficient AS LONG as it does not swamp Emporia’s server as more people start using and possibly abusing it.  And apparently the Sitesage device (designed many years ago) simply dumped an XML stream consisting of Voltage leg 1, voltage leg 2, followed by channel no and amps  to the Sitesage cloud server on demand, because when I made the same http port call, that’s what I got back.  My suspicion is that the Emporia device in my breaker box does something similar; when the server calls a specific port, it responds with a list of channels and power using XML or JSON, since that would be the simplest implementation in terms of putting software in the local unit rather than having IT do calculations and initiating communications with the cloud server.  So all I would need would be the port number to make the call to.

 

[chrwei]

sounds like Sitesage is a “pull” api, which is more difficult to traverse firewalls with.  Emporia uses a “push” api, which is much simpler for firewalls and offline caching.

I doubt it’s actually mqtt as well, mqtt doens’t handle streams so well, but handles state very well.

 

[helgew]

The Emporia device works differently from the Sitesage devices! It uses the mqtt protocol, as I mentioned above and does not wait for any server-side connections to retrieve data (i.e. it works only in push mode, not in pull mode). MQTT’s encryption mechanism, unfortunately, is more robust than most HTTPS implementations and not easily susceptible to man-in-the-middle attacks that could be used for a transparent proxy-type approach, for example.

Without the ability to configure the mqtt server to talk to, the Emporia servers (running on Amazon’s cloud platform) will always be needed. For some folks, that has advantages as those servers retain data much longer than the device itself could. For those of us interested in real-time monitoring, they aren’t really needed.

[helgew]

I doubt it’s actually mqtt as well, mqtt doens’t handle streams so well, but handles state very well.

It’s definitely mqtt, which is perfectly suited for this application! The data packets are fairly small (one number per channel and a channel id sent every second) and the protocol designed around unreliable networks situations often encountered in remote sensing applications.

Here is a bit of my device’s network traffic. 10.0.3.50 is the IP of my device and 8883 is the secure MQTT server port it is talking to:

09:20:50.623508 IP 10.0.3.50.61360 > ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883: Flags [P.], seq 443:886, ack 1, win 5596, length 443
09:20:50.696145 IP ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883 > 10.0.3.50.61360: Flags [.], ack 886, win 65535, length 0
09:20:52.624809 IP 10.0.3.50.61360 > ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883: Flags [P.], seq 886:1329, ack 1, win 5596, length 443
09:20:52.702914 IP ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883 > 10.0.3.50.61360: Flags [.], ack 1329, win 65535, length 0
09:20:54.641881 IP 10.0.3.50.61360 > ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883: Flags [P.], seq 1329:1772, ack 1, win 5596, length 443
09:20:54.713688 IP ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883 > 10.0.3.50.61360: Flags [.], ack 1772, win 65535, length 0
09:20:56.642053 IP 10.0.3.50.61360 > ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883: Flags [P.], seq 1772:2215, ack 1, win 5596, length 443
09:20:56.715170 IP ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883 > 10.0.3.50.61360: Flags [.], ack 2215, win 65535, length 0
09:20:58.630211 IP 10.0.3.50.61360 > ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883: Flags [P.], seq 2215:2658, ack 1, win 5596, length 443
09:20:58.703876 IP ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883 > 10.0.3.50.61360: Flags [.], ack 2658, win 65535, length 0
09:21:00.626028 IP 10.0.3.50.61360 > ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883: Flags [P.], seq 2658:3101, ack 1, win 5596, length 443
09:21:00.699799 IP ec2-52-15-119-124.us-east-2.compute.amazonaws.com.8883 > 10.0.3.50.61360: Flags [.], ack 3101, win 65535, length 0

[chrwei]

are you sure it’s actually MQTT and not just something that looks similar?  I’ve done MQTT and for constant data streams it’s either very complex to manage or you’ll end up missing data.

an SSL encrypted stream would make a lot more sense, and at its core that’s what MQTT is.  it’s all the topic stuff that makes it difficult to manage.  if you don’t have a server fast enough to pull the data from the topic before the next datapoint hits, you lose.  It’s double the overherd vs a single TCP server with no benefits.

[chrwei]

port 8883 is not enough to make it MQTT.

[zerog2k]

It’s mqtts protected w/ mTLS (using a client certificate).

It’s AWS IoT MQTT implementation:
https://docs.aws.amazon.com/iot/latest/developerguide/mqtt.html

Anyhow, it should be totally possible with ESP32 or even ESP8266 to expose simple, local json/rest api, or be able to configure unit to additionally publish to mqtt server of user’s choosing. Of course the devil is in the details of how you expose, validate, manage, and secure this configuration lifecycle to users, but it’s all possible. Likely (obviously) just low priority by Emporia product/dev teams.

Alternatively, they could expose a proper “cloud” api, i.e. developer apis. But this would also be some amount of work to manage user api keys, protection (security, rate limiting, etc), configuration, etc.

Again, some amount of work, but in this day and age, any type of “cloud” service like this, having such an api and user access to their data programmatically (for allowing integration with other cloud/local services, e.g. home automation, etc) is considered “table stakes” to put it in product/biz-speak.

 

[helgew]

port 8883 is not enough to make it MQTT

Technically true, but a) Occam’s razor, and b) I was communicating with Emporia staff directly early last year when I first wrote my own API client and they did confirm that they had discussed firmware modifications to allow for publication to other MQTT servers. I do not know if the current firmware allows for over-the-air updates, but that would obviously be a requirement.

it’s all the topic stuff that makes it difficult to manage.  if you don’t have a server fast enough to pull the data from the topic before the next datapoint hits, you lose.  It’s double the overherd vs a single TCP server with no benefits.

I think there are some misconceptions about the MQTT protocol in the above. Messages to the same topic do not overwrite older messages unless they are both flagged as to be retained. Messages pertaining to a topic that has subscriptions using a persistent session are retained by the broker until they are retrieved. Quality-of-Service flags can ensure that messages are received on the server before they are discarded on the client. MQTT has very little overhead compared to other “standard” protocols like HTTPS. Obviously, one could design and implement a task-specific and scalable protocol with a lower overhead than MQTT, but why re-invent the wheel?

[duunoit]

@kpnobvious

Could you elaborate on:

1. I had to modify the sample JSON to get pyemvue to tag properly, or else everything showed up as the channel name\# from Vue.  I can provide a sample if needed.

Thanks!

[duunoit]

that was supposed to tag @kpnobvious3

[kpnobvious]

hi @duunoit

Sure, I realized the change i was talking about in #1 was in the vuegraf.json file not in PyEmVue.  To that end my vuegraf.json file looks like this in compared to https://github.com/jertel/vuegraf/blob/master/vuegraf.json.sample

Line 12 – name of your account, i believe this is done during setup
Line 17 – name of your Vue (under Device Management, Home & Circuits, Vue Info)

After my last channel name, delete Jertel’s 2nd panel, so delete lines 27 thru 39.

You should be all set at that point, if it runs and the tagging doesn’t work, the json file is misformatted.

Thanks
Mike

[duunoit]

@kpnobvious3

 

Thanks Mike.  I got it worked out.

[drewbeer]

i’m going to be a little honest here. i really like your product, but i’m concerned its going to turn into a subscription model, the devices are great, and i want to buy one, but like many others on here, i’d rather not pull day from you and cost you more money, i’d rather just post to my mqtt, or api that i can collect data with.

i have a rainforest energy monitor. it posts to my internal api that i then clean up and dump into mqtt. however SCE (our power company) has discontinued the HAN monitoring and my meter was just replaced because something failed in it. I’m looking for something new, and either i go some open source solution or i something more professional.

i’d really like to see alternative or additional mqtt servers supported.

thanks

[nrus2222]

I really want to dive in with this device, or another that meets my needs for smartthings integration. Looking at Sense, Eyedro and Emporia, only Eyedro has ST integration, but only Emporia has dedicated circuit monitors – I really want the ability to trigger ST actions based on current thresholds on individual circuits, preferably with sub 1 minute lag (and prefer not to have to rig something with dedicated individual sensors on each device, as well as benefit from the power usage data).  Anyone in this community have a suggestion for me?

Unfortunately, I lack the skills to contribute, and fine with the limitations of pulling data from the cloud.  I hope someone is successful in developing such an API, and either this community or the company can create a DTH for ST with this kind of capability.

As someone said earlier, I know there are “dozens of us” who want this type of thing, so likely not so motivating to the company, but I’ll be at least one more customer as soon as the capability exists with one of these energy monitor devices. I was encouraged by Emporia’s comment on another thread that this is “on their radar”.  I will continue to lurk on threads like this until then!

 

[kpnobvious]

Hi @Nrus2222, Magico posted something earlier, take a look here:

https://github.com/magico13/ha-emporia-vue

[ehinkle]

Looks like an interesting product I would like to buy but think I will wait till more traction is in place to have a way to pull data locally from the device. Had purchased some iotawatt system to see how well that works.

[nrus2222]

Thanks very much KPN and Magico! It looks like exactly what I need for Smartthings.

Home Assistant has not been on my radar, although I seem to remember a similarly named product from the old X10 days.  I’ve got a lot already invested in Smartthings ecosystem, but I have been very worried about Samsung’s apparent plans to abandon their Groovy API, and leave users like me with multiple custom DTH’s and my WebCore integrations behind.  Until they pull the plug, I’m really not ready to change everything.

I will definitely investigate Home Assistant for a future pathway (for all my HA if Samsung pulls the plug on me). I wish I was smart enough to turn Magico’s github contribution for HA into a ST DTH.  I still hope somebody else is working on this!

[Ted @ Emporia]

Dear Customers:

We appreciate the enthusiasm for Emporia devices but a minority of customers are causing a majority of the load on our servers.  Some users of the PyEmVue library are requesting years of Monthly usage data every 2 seconds.  That is unnecessary since previous month data won’t change and current month data only changes every hour.

We aren’t cutting off PyEmVue completely, but we plan to introduce limits in the cloud to prevent a few users from overloading the cloud.  In the meantime, please be respectful by fetching data less frequently and avoiding re-fetching unchanging data.

We also intend to change the app to have it fetch data less frequently when the app is left running continuously.  We want to present up-to-date data when it is helpful but we need to avoid fetching data around the clock when no user is looking at it.

best,

Ted

[kpnobvious]

@tedworksatemporia

Thanks for posting and opening an issue on Magico’s github.

Do you think this is more related to device usage API requests?  Maybe if there is no end time specified, it returns all values?

url = API_ROOT + API_DEVICES_USAGE.format(deviceGids=gids, instant=_format_time(instant), scale=scale, unit=unit)

Or maybe a chart request with no correctly formatted end value?

url = API_ROOT + API_CHART_USAGE.format(deviceGid=channel.device_gid, channel=channel.channel_num, start=_format_time(start), end=_format_time(end), scale=scale, unit=unit)

If you have the ability to post say the top 5 or 10 public IPs making the calls to AWS, that might help some users figure out if they are the problem.  I think both Jertel and @helgew use pyemvue but may make their own posts to get the data

https://github.com/jertel/vuegraf

https://github.com/helgew/emporia-downloader

Also is there any progress on sending data locally say MQTT or maybe helping magico with some API documentation similar to what Flume has published for their water meter? https://flumetech.readme.io/reference

As I’m sure you’ve read from many comments in this thread, if there was a local option, there would be more customers and a lot less cloud polling.

 

[jj613]

In response to the last message from Ted about server load:   It’s reasonable to impose request limits on API users.   If you published your API instead of letting people reverse engineer it as they have, you would undoubtedly have published limits and best practices.  If you allowed the devices to work off the cloud, the heaviest users might get off your servers entirely.   And if you didn’t limit one-second and one-minute data in the app the way you do, there would be less reason for people to be sucking data out of your servers to begin with.   I’m not doing it yet but that’s the main reason I WOULD do it … so that I have permanent access to granular data.

I really think you’re missing a few SIMPLE tricks that would make a lot of your customers a lot happier and open up new uses and value for the Vue.

[wz2b]

 If you allowed the devices to work off the cloud, the heaviest users might get off your servers entirely.

I agree with this, 100%.  I really hate this data going to the internet at all.  I only allow it because right now there is no other choice.

We also intend to change the app to have it fetch data less frequently when the app is left running continuously.

An API that has a parameter “Send all new data since time=160231015130515” is a pretty typical way to do that.  The servers should be smart enough to be able to quickly determine “there’s no new data for that device.”  Even if you had tens of millions of devices deployed, that could still be cached on a t3.medium running Redis, it should be hours to days of work to accomplish that level of scalability, not weeks or months.

 

I really think you’re missing a few SIMPLE tricks that would make a lot of your customers a lot happier and open up new uses and value for the Vue.

I cannot overstate the importance of what you just said here.  Saying I agree with it 100% does not adequately convey how strongly I agree.  It’s true that open systems like we’re asking for directly effect a small number of enthusiasts (hackers, nerds, whatever you want to call us).  But from Emporia’s perspective that’s not the important part.  The importance is that happy integrators and open source developers turn into evangalists and from a marketing perspective that’s exactly what every company needs.  Companies don’t need b.s. fake “viral” marketing campaigns at all (believe me, we see through those) if, instead, they have an actual community of enthusiasts evangelizing on their behalf.

This stuff is very, very important.

 

 

[kpnobvious]

Although Ted never replied in this thread, there was a decent discussion in github and Jertel and Magico already made changes to help reduce some of the requests.  So that covers two out of the 3 tools typically mentioned in this thread.

https://github.com/magico13/PyEmVue/issues/19

[helgew]

I think both Jertel and @helgew use pyemvue but may make their own posts to get the data

My downloader predates pyemvue and is written in java. It does not use pyemvue at all, which is written in python.

The API already includes parameters to enable someone to download only a subset of data and if you run my downloader continuously as intended, it will be “gentle” by default, i.e. request a chunk of data every 5 minutes and only request data for those past 5 minutes. Of course, someone could potentially run it differently and re-download all historical data every second, but that is rather difficult to prevent on either the client side or the server side.

• This reply was modified 3 years, 5 months ago by helgew.

[kpnobvious]

Thanks @helgew!

I updated vuegraf’s code manually since i had made some customizations to it.  I also updated pyemvue as there was a bug noted in there.  I would suggest everyone else update as well.

I haven’t noticed a change in my graphing, but have seen a significant decrease in data point collection

The data points i was collecting dropped from about 4500 over 5 minutes with the older per second collection to 80 with the per minute collection.

[wz2b]

The API seems to give you an arrow now if you request more than 1 hour of 1SEC data at a time.

[5310]

Hi

Forget the API stuff..

All they have to do is allow their APP to use a local MQTT broker address instead of theirs during the APP setup. This will stop over loading their servers and make everyone happy. Everything is being sent via the  MQTT protocol. Ten lines or less of code would fix it, I bet.

And understand NO SUPPORT needs to be provided. You use a local MQTT broker and you’re on your own. ‘PERIOD’.  If you don’t know how to setup a MQTT broker server then this isn’t for you. Because that the only way this can work.

Marty, please consider this when you update the APP next time and I’ll be happy to tested it.

[wz2b]

All they have to do is allow their APP to use a local MQTT broker address instead of theirs during the APP setup

How do you know it’s using MQTT, did somebody confirm that?  I suspected that was the case but was hoping for confirmation.  Checking my NAT table:

ipv4 2 tcp 6 7395 ESTABLISHED
src=192.168.0.126
dst=18.189.226.239
sport=60459
dport=8883

Port 8883 is the IANA registered port address for “Secure MQTT”.  So it seems likely.  I’m sure it’s doing some kind of negotiation for the TLS session.  That COULD be using certificates but it probably doesn’t expect a server cert so it seems to me you could do two things

1. Set up your own local MQTT server
2. Do either some DNS trickery trickery to just redirect it to yours, or some iptables trickery to do the same

Regarding “…then you don’t need an API” I don’t agree with that.  Some people will be fine with this data going to somebody else’s service.  I would still do both.  When it comes to order of importance, having a redirectable API would be better for me personally but I suspect we’re in the minority here when it comes to customers.

 

 

 

 

 

 

[5310]

Hi wz2b

I can confirm it’s using MQTT on port 8883  and user / password to authenticate.  With the addition of a local address instead of their you’ll have full access to all the data being collected every second with no cost on their part.

You are correct too, we are not their targeted customer but we would for sure put more money in their pockets than it would cost them. Its a win-win. I think SUPPORT is the issue. They can’t support us but I wish they would see the potential.

[wz2b]

They can’t support us but I wish they would see the potential.

Strangely, I don’t think we’re all that difficult to support … we more or less just need to be pointed in the right direction.

[5310]

Really it a support issue.

Think about it. Their current customer does everything from APP.  Load the App, setup your device and start receiving your data. No technical issues.  No technical support. So if your a DIY user you do it all, either a local MQTT broker or something like mqtthq.com.  Since they aren’t getting the usage data to aggregate they can’t make any additional revenue.  Get it.

[wz2b]

Redirecting through DNS was easy enough but mosquitto doesn’t like something.  I’m not super familiar with mosquitto so I set it up like this:

certfile /etc/xxx/cert.pem
keyfile /etc/xxx/privkey.pem
port 8883
protocol mqtt
tls_version tlsv1.2

The Vue2 is definitely trying to connect to it but something is failing, probably the TLS exchange.  Maybe the Vue2 is smart enough to check the server cert and make sure it’s valid?

On startup, the Vue2 does three DNS queries

• a2poo8btpqc3gs-ats.iot.us-east-2.amazonaws.com  this is the MQTT destination
• fwsrv.emporiaenergy.com   this is probably to check for firmware upgrades
• pool.ntp.org   network time server

Not sure what’s with the wacky hostname, but the DNS response is a list of 8 addresses so they’re doing some load distribution.

Anybody who knows mosquitto ahd has any tips please let me know.

 

 

 

[wz2b]

I upgraded mosquitto and it changed:

1624935763: New connection from x.x.x.x:52059 on port 8883.
1624935922: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
1624935763: Client <unknown> disconnected due to protocol error.

 

• This reply was modified 3 years, 5 months ago by wz2b.

[wz2b]

Could the Vue2 be using a client certificate?  If that’s the case, I think my mosquitto doesn’t know to trust it … that’s what the error seems to indicate, anyway.

 

[7_2kW_in_Mesa_AZ]

Hi all-

I will admit I am here for some serious handholding (sorry in advance).

1. I am wondering if PyEmVue would allow me to retrieve minute interval data that is more than 7 days old, that is as far back as I can go in the app.

2. Could I get some help getting this up and running?

a. I have a clean Ubuntu 21.04 VM up and running with Python 3.9.5

b. I have run python3 -m pip install pyemvue and got all the dependencies

c. how do I use the command now?  Anytime I try Python says it can’t find the module.

 

Thank you all

[5310]

hi wz2b:

The certificate needs to be created by you on your MQTT broker server. You can build one using Lets Encrypt.  It will create the .pem files but you will need to rename or copy them using names in the config.   Also add a user to your MQTT broker using the same user/password that was used to setup the VUE And also don’t forget to restart your MQTT broker. Then use a MQTT Client, like MQTT Explorer, and connect to your MQTT server with the user/password you created.  If you get this far try subscribing to  ‘prod’ topic.

[helgew]

The certificate needs to be created by you on your MQTT broker server. You can build one using Lets Encrypt.

I don’t think this is as easy as described, because the CN of the certificate would have to be the same as the VUE is trying to connect to.

[5310]

Hi

You have the certificate for your MQTT server.  Its concerned that the connection is secure not who it is. It will use your ISP Internet address, a port forward to 8883  and duckDNS type setup.  Google duckDNS setup. Really not that hard. You can always get a domain and create the forward from there.  I use Cloudflare and it’s less than $10 a year. Then do the forwarding inside my router so when it gets emporia.site.com it forwards to a Lan side address on port 8883 which is my MQTT server.   Many different ways to accomplish this.

[helgew]

Have you actually tried this? A good SSL client most definitely would care who it actually is connecting to, otherwise man-in-the-middle attacks would be much easier to pull off.

Also, you are assuming that the VUE doesn’t use a client certificate to connect to the server, which would have to be signed by the server’s private key. Do you have any evidence that that isn’t the case?

[wz2b]

These are good questions you asked.

For my web server I use letsencrypt, so I tried starting mosquitto pointing toward the latest letsencrypt certificate and that worked – at least, i was able to pub/sub from a mqtts client without any errors.  However, that’s a legitimate certificate, so there’s still the possibility the vue is looking for something with a CN that matches the domain.  I wouldn’t THINK so though based on the wacky hostname, which looks like an aws auto generated host.  Or it could be looking for a specific certificate.  The answer to the question is that I’m not really sure and am trying to figure it out.

 

Also, you are assuming that the VUE doesn’t use a client certificate to connect to the server, which would have to be signed by the server’s private key. Do you have any evidence that that isn’t the case?

 

I think so.  I receive this log:

OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

There’s a little information out there about this message, but I’m not familiar with mqtts or tls to really understand what OpenSSL is trying to tell me here.  I interpret this to mean the vue2 is sending a client certificate but it doesn’t match any known CAs so it is rejected.  One thing I tried was forcing mosquitto to only accept v1.3 or later and this produced a different error message (invalid/unsupported TLS version).  So my conclusion based on that is that the Vue2 is indeed sending a certificate and it’s TLS v1.2.

I wonder if there’s some way I can test this with something like ‘netcat -l’ but that accepts TLS.  I found something named ‘socat’ exists.  Maybe I will try that next, at least to see if there’s a way to tell it “Accept any certificate, just give me some logs to tell me what it’s trying.”

 

 

[zerog2k]

The devices seem to be based around Amazon IoT platform, which provides frameworks and tooling to handle a number of things like device provisioning/registration, including issuing client certificates, app/user identity with Cognito, and providing mqtt/http service endpoints to collect data and establish communication channels with devices.

The endpoints and ports you describe are most likely amazon iot mqtt endpoints. (Yes they will have strange names, as they are generated by AWS for some given service instance.) For more info see: https://docs.aws.amazon.com/iot/latest/developerguide/protocols.html

The devices are certainly using client certs issued and signed by some amazon ca or similar, see: https://docs.aws.amazon.com/iot/latest/developerguide/x509-client-certs.html

Usually, there is some way to make a TLS server ignore cert validity of client (for testing purposes ;), but in this case, I’m not finding such an obvious option: https://mosquitto.org/man/mosquitto-conf-5.html

 

 

[zerog2k]

another angle might be to add all the amazon root CAs listed here

https://www.amazontrust.com/repository/

to the ca cert file you pass to mosquitto configuration (in addition to whatever trust bundle is being passed now, which contains your server cert – I believe it will need to contain a trusted ca for both server and client certs to be internally happy)

 

[5310]

wz2b

Assuming that you are seeing the Vue trying to connect to your MQTT server, I would suggest to use the APP and try reset or add it again. I believe the certificate thing may occur during the initial setup.  And no I don’t have any evidence that this will work.  I would try but mine is installed and live.

[helgew]

I believe the certificate thing may occur during the initial setup.

That is not how SSL works at all. If the client uses a client certificate, this will need to be pre-installed and is used to authenticate the client to the server during every new connection made by the client. Conversely, the server certificate is requested by/sent to the client on every new connection as well to verify that the server is indeed the correct endpoint to connect to. Only after both server and client have established that the other is who they are claiming to be (the TLS handshake phase), they will negotiate the encryption protocol to use and exchange the necessary information.

Using a letsencrypt-generated certificate with the domain name of the internal MQTT server results in a failed TLS handshake with the VUE. Using a self-signed certificate with the domain name the VUE is trying to connect to results in an “unknown CA” (CA = certificate authority) error.

• This reply was modified 3 years, 4 months ago by helgew.

[magnetus]

helgew
have you ever release a windows version of your emporia-downloader?
thanks
magnetus

[EwertonMoreira]

+1 vote for the MQTT option!

I got Helgew’s code running on my raspberry pi but I still didn’t get my device. Does anyone have the credentials for the demo account?

 

[Eric12312312]

@Emporia

If you want me to buy your device, let me collect my data locally. If your product would let me leverage my own MQTT system, or EVEN had a serial output over wifi, I would have one in my house already.

A lesson to learn- people who are serious about home automation, and data collection, REALLY enjoy having local control over their devices. My philosophy- if the device doesn’t work when the internet isn’t up- I don’t want it.

Give me the ability to use your device, without a connection to the cloud, and I will buy one.

• This reply was modified 3 years, 3 months ago by Eric12312312. Reason: enable notification email

[5310]

The simple answer is they don’t want to, either because of support issues or loss of revenue from your data, because it would simply take adding local entry fields to the Emporia App.  They need to add local address, port, user and password so the device would connect to your MQTT broker instead of theirs and provide the Topic schema.  That all it takes.  Even of they could sell ten times the devices, I don’t see them doing it.  So that tells me the device isn’t that profitable so that data must be.

Just my opinion….

 

[5310]

 

Done hold your breath.  Yes, it use an ESP32-WROVER-G device,which I believe only supports 7 analog to digital sensors but what isn’t known is what else are they using for the ADC services.  The VUE v2 supports 19 sensors. Nothing is identifiable and without that knowledge your are probably not going to get any code from ESPHOME to work. ESPHOME needs to know what ADC and how it’s configured in order to  work. So good luck,

It would be so much easier for us if they’d allow their APP to provide provisions for entering MQTT Broker info.

[Eric12312312]

I do see it does have an auxiliary processor on board as well.

 

From a simple perspective, can sniff for serial traffic from the esp. If there is no traffic, then I can leverage my oscilloscope to find the function of the individual gpio pins.

 

From that point, it’s not terribly hard to get everything configured correctly.

 

But, the important part 8s figuring out what method it uses to multiplex all of the sensors. My initial guess- is that the auxiliary processor handles a big part of it….

 

Oh well, a few evenings scoping will shine some light.

 

Or, ya know, they could allow local api access…

[Bruc]

@eric12312312 – lets talk more about osci.. (crlogic at ymail dot com)

[John Polasek]

An api that could return a JSON or XML response from the Emporia cloud, while not ideal, would be sufficient for me;  I am currently pulling download CSVs once a month or so to analyze, and have a plugin for the Indigo server that can request and read either format.  The only thing is that to be really useful for real time control, I would have to be making a request a minute, and (as others have implied) if EVERYBODY starts doing that, it could easily swamp your servers… which keeping the access local would not.  I will say that as soon as I CAN get the ghostXML working to read the real time data reliably cloud or local, I will likely be purchasing a couple more to equip my sister’s and mother’s homes as well.

[Eric12312312]

Meh, I went in the direction of a open-source competitors project.

Native LOCAL integration with my home automation. Supports REST, Serial, JSON, MQTT, You name it.

I don’t have to worry about the company going under in 5 or 10 years, because the device runs my own firmware….

And, I am not at the mercy of somebody else’s web services.

Did I mention, I get data updates as quickly as I want. Every 1 second if I need.

[that_kid]

@eric12312312 Who did you go with?  I need to look in that direction as well

• This reply was modified 3 years, 3 months ago by that_kid.

[Eric12312312]

Well, in before they just ban me, Lets just say- Circuit Setup without a space.

If they do indeed ban me- I dont’ feel bad.

Based on the official replies in this thread- They don’t listen to customer needs.

• This reply was modified 3 years, 3 months ago by Eric12312312.

[that_kid]

I’ve been thinking of doing the same.  I was hopeful that a local option would become available but it doesn’t look like that’s the case so I’ll have to return this unit and go another route.

[Eric12312312]

Its quite sad too- The product HAS the capability.

They literally have the perfect hardware stack for this. ESP-Based, Modular, Cost-Effective, and great form-factor.

They just choose to purposely force the use of cloud-ONLY apis on everyone.

If- they would literally just give a simple option, for any local access- whether it be- allowing a user to specify a local MQTT server to also send updates to, or an API, or, user-replaceable firmware (without having to open the case)- I Literally know an entire community of thousands of people who are actively trying to invest into a tool with these exact capabilities, but, with local access to data.

But- being a developer / hobbyist hardware person- I do feel these units are being sold at a loss, or at least- not at a profit. Components are not free. CT Clamps cost money. And- based on the official responses- I can only assume they are somehow profiting from the cloud data, which would backup their reasoning behind a cloud-only API.

Personally- I actually had one of these devices ordered, with the intentions on popping the case off, and reflashing the onboard ESP to run my own custom firmware. The reason I backed out- is because I could spend 50$ more, and not have to worry about ANY of those concerns, and I would receive a device, with documentation on exactly how to run my own firmware, without any issues. It’s all completely open source.

For reversing this device- reversing the logic behind the ESP32 would be piece of cake. Tracing out the GPIO/ADC ports is quite easy. Determining the purpose behind, and reversing the auxiliary processor would be a decent amount harder, but, still doable. All said and done- it would likely take me 3-20 hours to properly reverse this device to run my own firmware. Or- pay 50$ more and have a solution I can put into place, in minutes.

BUT- for emporia- we have voice our concerns for wanting to use our data locally without your cloud api. You refuse to listen- so, Congratulations on missing out on a unique opportunity to cash in on quite a few thousand home assistant users looking for energy monitoring solutions. (This month’s update added a ton of nice fancy energy monitoring features)

 

[that_kid]

It’s interesting I’ve been looking at that other product for months and when the new HA update came out I decided it was time to get one.  Then I saw the vue2 on Amazon and was like wow that’s great price point and the fact that I can wire it directly to a breaker and have everything be self contained with that number of inputs it was a no brainer.  Just like you I had the same thought about being sold at a loss and also about re-flashing.  When I got my unit and discovered it uses MQTT I thought it would just be great if I could use my own MQTT server to ingest everything and not have to reinvent the wheel.  Some hours later I’m seeing that I should’ve just went with my initial order and saved myself some hours of “discovery” and disappointment.  It’s funny because a friend of mine sent me an amazon link to the vue 2 this morning saying he was thinking of getting one to use with Homeassistant and I had to redirect him elsewhere.

[jj613]

I don’t care if they want to profit from my data.   I use lights at night, A/C when it’s hot, you know, it’s really not that private or that interesting.   If I had an illicit grow farm or crypto mine I obviously would not like this solution.  But I don’t.  For me, there are app features I’d like to have … and I accept that they have a roadmap and can’t do everything at once … but they could let their community build the features.    IMO the biggest trick they’re missing is *power* (as opposed to energy) monitoring and management.  You need much more granular data to do that, and you need to be able to do it off line.   The granular data is there …. they just throw it away.  🙁

[Eric12312312]

I cannot provide a link, as that seems to be the only way to invoke any official response here-

But, I can say- if you look at /r/homeassistant

There is a thread I created, titled “Home Assistant – Options and Solutions for Energy Monitoring”

I documented every major, well-known option I could find, along with pros, cons, and if it had an integration with Home Assistant, rather official, or third party.

There are options ranging from 50$ up to 399$ Ranging from a single circuit, up to 42 circuits.

[that_kid]

I see the heading but not the rest of the post

[Eric12312312]

xtremeownage

.com

/2021/08/11/home-assistant-energy-monitoring-options

[nrus2222]

Time for me to unfollow this thread. Enough BS noise in my inbox.

Honestly, what fantasy land do you geeks live in that you think this is an effective way to convince Emporia to support your hobbiest whims?

99.9% + of their users don’t give a rat’s ass about any of this.  I was just hoping to find a Smartthings DTH for this thing, not spend days writing code and setting up servers, etc.

A small but vocal group of douchebags who have way too much time on their hands say what?

 

[wz2b]

A small but vocal group of douchebags who have way too much time on their hands say what?

I think that’s pretty harsh. ” You won’t get anywhere by beating Emporia up” would be a fair point.  But I think it’s a mistake to discount the advertising value of a small group of technical evangelists.

[Kevin @ Emporia]

I would just like to throw in some personal notes about the direction that this thread has taken. I think everybody here raises valid points. As an advocate for open source software myself (who regularly contributes to OS projects), I would love nothing more than to have a DIY/hackable device to work with my local data. I also understand the business perspective of managing it from the cloud. As some have pointed out, our goal is to build an easy-to-use quality energy monitoring solution at the lowest possible price point. The majority of our user base has no concern to monitor their device information locally (even if they had the technical experience to do so), but we also have the veteran users who need to track individual watts. Its difficult to find a good balance between keeping the ecosystem simple to use while providing the vast amount of functionality that could be possible. We also have to juggle development resources between hardware, firmware and software priorities. Our company is small, employee run and financed, and we’re in the process of launching our two largest hardware products to date. A lot is happening on our end, and we’re just hoping to provide a decent quality product with a quality customer experience. It’s unfortunate that we can’t live up to each users expectations for how our product should behave, but that is the world we live in.

If possible, I’d like to keep this thread open and focused on how to use the existing capabilities (official or un-official) to access your data.

I’d also like to point out that while the Emporia team might not be very vocal on these forums, we certainly read every post and many of our employee’s read the forums on a regular basis. We like to keep this space for customers to interact with one another and we feel like if Emporia help is needed, then we’d encourage you to reach out to the support team directly.

(I’d also like to clarify that we’re not selling anybody’s data, all of our revenue is driven directly by the purchase of our devices. Just a heads up.)

• This reply was modified 3 years ago by Jim @Emporia.

[John Polasek]

I will say that I have to agree with Eric about the desirability of local access;  I had forgotten about it because it happened 6 years ago, but I got burned about $500 when I bought 4 WiFi soil moisture sensors that I tied into my home control system through their cloud based API and used to skip watering on my sprinkler system by shutting off power to the solenoids until the moisture in the zone when below 20%… and 8 months later the company folded and their website went 404…

 

[schanders]

@eric12312312 , you and @itsjohannawren should talk. Please add me to the conversation as I would like to take a look at a custom firmware schanderbally at google mail

[XplodingData]

I have been considering the Emporia products as well for a few weeks and have been googling to find a way to do data collection locally.
After reading this thread I have to jump in and agree with the other posters that local data options would really really help sell your product, especially since you have said your only source of direct income is from the sale of the product, and that you don’t sell data.
So I have to question.   How much can you possibly make off a kit that’s only $110CDN to buy?   How much is each customer projected to cost you per year?  How many years will you be able to continue to support and keep the product alive?  If I buy one, and go through the hassle of having it installed, will you guarantee i get 10 years from it?

I think you’re really underestimating the number of units that could be sold without having to support monthly server/support costs.
If ongoing customer support costs are a concern with DIY people, Why not consider a one-way firmware upgrade to Open Source firmware that removes the device from your cloud and have it be officially unsupported?  Heck, you could probably crowd fund the development costs.   Your product looks fantastic and professional, I’d happily donate $10-20 to a crowdsourced development goal.

But sadly, now I’ll be purchasing a different device.  So put me in the column of lost customers.

[Alex]

I just registered to this forum t0 jump in as I agree with the others posters looking for local access or at least a good API.

If cloud is critical for Emporia’s business, keep it mandatory and provide a dual path with local access to limited data at the same time. I don’t mind Emporia looking at my data. They probably need a lot of customers to get to the AI side of things soon and start using the power of the cloud and data aggregation.

I am also in the HomeAssistant ecosystem. As mentioned a few post above, a big push is in the Energy Monitoring with the latest 2021.8 update. I had the Vue 2 ready to be ordered but pulled due to lack of API and local access. I was also burned with similar “cloud only” products (e.g. TrackPin, GooglePhoto, etc).

I get it that most people want it to be simple and easy and that the app and the cloud make all that possible. However, how many people are “nerdy” enough to want to monitor their energy usage and take the time/effort to install such a system in their electrical panel? If you really want simplicity, look at the competition with AI and two CT clamps… For the ones who are ready for 16 CT and a mess in their panel, they are probably also looking at other home automation products.

Emporia will win in the market place by keeping the cloud connection by providing new features and functions (bunch of smart things) using the cloud and integration with Utility providers. But they can also win with the Home Automation nerds by providing them with the basic data for their HA systems. Just provide us the “Current” from each CT clamp and “Voltage” from each legs and let us deal with the rest. We will then use your app and the cloud for the cool new features you will introduce and we will use Home Automation or other system for integrations and automations.

Finally, please speed up the delivery of the official API. That may be enough to cool us all down.

• This reply was modified 3 years, 3 months ago by Alex.

[zerog2k]

Exactly what Alex said. This is a business story for you!

The geeks and nerds who play with this stuff are your free advocates to the less techie friends, family, and coworkers who would buy into more of your products & services.

[Eric12312312]

/shrugs. Maybe one day they will listen to us.

Too many of us have been burned by cloud products going under, samsung depreciating old smartthings hubs which were perfectly functional… cloud vendors yanking away APIs (*cough* depreciated works with NEST, Blink, Ring…. ETC)….

At which time they hear the concerns, and just give us the simple ask, of local control- they will pick up a entire new business segment.

Personally- I just spent nearly 300$ on an open source alternative, which gives me the ability to run my own firmware, and complete local control. When the device dies, I toss enough ESP32 on the board, reflash it, and keep going.

That is 200$ more than this product sells for, for almost exactly the same functionality.

Just saying…..

[nateguchi]

A local API would be AMAZING. Even if that is just a UDP one. It really confuses me as to why Emporia is one of the very few manufacturers of energy monitors that doesn’t provide a local API

[blahblahblah]

I’ve got a suggestion :  Provide technical details of the Vue2 circuit board and signals to the ESP32 controller.

If Emporia aren’t monetizing the data in the cloud, it would be awesome if they share some technical information about the circuit and signals to the controller.  I can see that it has a ESP32 and a secondary controller (presumably to multiplex all the CT channels).  With some information on which pins connect to what and how the multiplexing works, the community could easily create custom firmware to meet our needs.  It would be easy to solder on some Swiss machine pins to have ready access the serial interface of the ESP32 (I cannot confirm nor deny that I’ve already done this).

Once any customer starts soldering on the board and installing new firmware, they’re not going to have any reasonable claim to warranty or technical support.  So, by simply providing some documentation, Emporia would be making many of the technical enthusiasts happy and not have to spend development resources on a local data API.  Plus, any semi-technical customers that want to go the custom firmware route would be helped by the community.

Once the custom firmware gets going and word is out in the home automation community, it will only help sales.  (And it would build a lot of good will with the enthusiast community!)

[sutekh]

I’ve had several of their devices sitting in my Amazon cart for months. I’d love to install them in my home (4 x 16 sensor devices) and my cabin (1 x 16 sensor device) and send data to my local InfluxDB / Grafana instance, but I’m not buying anything until they provide a local access API. Emporia’s now two plus year old assertions that “it’s in our backlog” or “we’re just a small team” is no longer believable.

Emporia: You could make this happen ASAP if you wanted to (don’t even offer support!) You’ve chosen not to. I suspect the rationale there is that you’re holding the door open to eventually start charging for access to the data. You’ve got threads full of people on your forums dissatisfied with that position who’s all been burned by abandoned vendor technology and it’s costing you business. You make a niche product that appeals to nerds and techies. It would be a mistake to assume we represent an insignificant portion of your potential market opportunity.

[blahblahblah]

Here’s an excerpt from the reply that I received from Emporia Support when I asked about providing block diagrams of the hardware so that custom firmware could be written (and save Emporia from doing the work to provide local data access themselves).  Emphasis mine.

“…  Our focus is to be a cloud-based distributed energy management company. Monitoring is the first step in that direction and the reason we’ve been building hardware/devices. The next part is the management functionality itself and how to control/optimize energy consumption and production. We’re in discussions currently about how opening up an API would affect this aspect of our future plans. Some argue that wouldn’t hurt revenues at all (or even help) while others are concerned that allowing our devices (currently sold at cost) would provide a hardware platform for other management companies to control. The main confusion I think is whether or not we’re considered a hardware/device company or an energy management company. Our vision is of the latter, but many customers consider us as part of the former.  …”

So, if Emporia are really selling hardware at cost and they are not monetizing their customer data, then the next logical conclusion is that Emporia plan to start charging for their cloud services.  Maybe that will be in the form of some premium service while still offering a basic, free cloud service.  Or maybe it will be a bait-and-switch where they start charging everyone with their hardware installed in the hope that most customers would rather pay than remove the (otherwise useless) devices.  You can draw your own conclusions.

I bought a Vue2 several months ago and have been working to reverse the firmware and hardware.  I’m not a hardware or software engineer so it’s been a slow process but every step has been a learning experience about IoT devices and electronics in general and I am enjoying the journey. I haven’t installed the Vue2 yet as I want to keep it in factory-fresh / unprovisioned mode while I work on the reversing.

There are others who have made excellent progress on this front.  I won’t link to them here but they are easily found with a quick search.  If your device is unprovisioned, there is an option to modify some config data in the device so that it sends data to your own local MQTT server without needing to replace the firmware entirely.

I’m disappointed in Emporia’s general apathy to the hobbyist community but with their response above, I know that there’s no point in waiting for them to provide an API or method to collect data locally.  They’ve given me all the motivation I need to see my reversing project through to the end.

Updates to follow.

 

 

• This reply was modified 3 years, 1 month ago by blahblahblah. Reason: Fixed mangled HTML

[Eric12312312]

/shrugs. I have even clarified, I would spend MORE for a product which gives me full control over its software.

This year, I would estimate I have dumped 600$ on purchasing various open source alternatives, such as circuitsetup and iotawatt to evaluate and review.

I made my goals very clear earlier in this thread…. I will not be buying any product, which has a dependency on the cloud which is not under my control. I am sick and tired of having cloud apps decide to yank access or change TOS causing these products to become another piece of e-waste.

 

[zerog2k]

It’s like the old saying, if the product is free, then YOU are the product.  In this case, they are claiming that the device is sold at cost, so essentially subsidized, meaning again that you and your data is the product/ business model.

However, looking at these devices, the hardware BOM is probably 30-40 bucks at most, and they sell for like 100. This is before the engineering and supporting costs around developing hardware. The real cost is probably the cloud services bill around running their platform, plus the cost of developing and maintaining the app. So maybe “at cost” is believable, but if you don’t run on their app or cloud platform, then it kinda takes those parts’ cost out back of the equation. So I seriously doubt they would ever entertain a cloud-free operation, as they would lose the data access, which I feel is where the value is from

 

• This reply was modified 3 years, 1 month ago by zerog2k.

[Eric12312312]

You are missing the cost of the CT Clamps. Those add up pretty fast.

While, they have claimed they do not sell data, I personally do believe customer-data is apart of their business model. Otherwise, they wouldn’t have any incentive to not assist us in leveraging local access.

We KNOW a simple configuration change will direct the device to a local MQTT server. We KNOW the device is based on a easily flashable ESP. However, we also KNOW the company will provide zero assistance to assist us with building a configuration for Open source firmwares which can run on the module- to the point they openly exclaim, cloud only. So…. I do believe the data is indeed somehow apart of their business model.

But- I also do believe they are telling the truth, when they sell the unit at-cost. The unit includes all of the CT-Clamps, and adaptors needed to get everything up and running.

[helgew]

I feel like y’all are going in circles here. Emporía clearly has a business model that does not include providing a local API. Complaining and speculating about this is pretty futile, in my opinion, but obviously everyone is free to decide how they spend their time.

I will no longer be monitoring this thread, so if anyone needs help with the emporia downloader tool, please contact me through GitHub.

[dekes1]

Emporia team, here’s another idea: partner with the Home Assistant team.  The recent addition of a full energy usage and monitoring stack in HA means that a whole new audience of hobbyists will now be interested in adding energy monitoring.    As disappointing as it seems that you will not be releasing a local API, what about building a native cloud integration with HA?

Work with them to build a supported integration with Energy Monitor and Utility Connect.  Advertise it as “Home Assistant” compatible.  Get added to their list of supported integrations.  The home automation industry is growing massively and HA is the “go to” solution for the serious hobbyist – leverage that community to spread the good will for the Emporia products.

[jtodd]

Putting my +1 in here – just found out about Emporia today, and will not be purchasing. Would have bought three, but I have a closet full of gear from companies that folded or no longer support their API, and I also object to that data leaving my property at all.  So no cloud for me on IoT, thanks. Wake me when a local query/collect method is available.   What mystifies me is that by opening up a local collection method they would sell a lot more units, since third party integrations would spring up without needing to be hand-held by their development team.

[flaviu]

Hey, if you’re particularly technically inclined, I’ve found a way to get it to output its messages over MQTT to a local MQTT broker and written a blog post on it. I haven’t attached it to Home Assistant or developed a step-by-step instruction guide on how to do this due to lack of time.

Here’s what it looks like when I push the data from MQTT -> InfluxDB -> Grafana:

[wz2b]

Searching the binary for that string shows us a section where the words “ssid”, “emporia”, “password”, “emporia123” are found in close succession.

Are you saying it attempts to connect to a local network named ’emporia’ as part of some debugging code, or as part of the provisioning process?  Maybe you’ve uncovered some factory test mode?

If it isn’t a virgin device, does it still look for the ‘Emporia’ network on start-up?

I’d still be interested in decoding the GATT message format, I’m just not very good at that kind of reverse engineernig.

[flaviu]

> Maybe you’ve uncovered some factory test mode?

Yup! That’s exactly what this is. And by setting TEST_MODE=1, it’s forced into the factory test mode forever, and never tries to carry out the provisioning process.

> If it isn’t a virgin device, does it still look for the ‘Emporia’ network on start-up?

I don’t know, but I wouldn’t be surprised if it didn’t. I’ve never connected my device to their app or network. However, I bet you’d be able to work around this if you used my NVS, combined with your pre-programed calibration data. The NVS is the main read/write location for the ESP32.

[jj613]

@flaviu that is more effort than I’m prepared to go to just to understand my energy usage.   But … it is heart warming to know there are still people in this world who are still so curious, creative, and knowledgeable about the entire tech stack from sub-chip level to app level.   Seriously I enjoyed reading your blog, you made me happy.  Thank you very much for sharing.

 

[flaviu]

@jj613, thank you, and I’m glad you enjoyed the read 🙂

The magic of software is that the hard part only needs to happen once. After that, it’s “buy $5 of stuff off ebay and click the button” for everyone else. We’re not there yet, but it’ll happen.

[taztaz]

Hello @kpnobvious3
can you share the script related to https://community.emporiaenergy.com/topic/api-to-pull-data/#post-7025
that allows you to generate the json please. I am very interested

• This reply was modified 2 years, 12 months ago by taztaz.
• This reply was modified 2 years, 12 months ago by taztaz.

[err]

People should checkout @flaviu’s github.  Full ESPHome support and it works great!

https://gist.github.com/flaviut/93a1212c7b165c7674693a45ad52c512

[flaviu]

> People should checkout @flaviu’s github.  Full ESPHome support and it works great!

Thanks for calling this out! There’s probably still going to be some changes between the current state of things and a “finished” version, but it absolutely does work! It’s very exciting to say that this hardware no longer requires the cloud.

Better installation instructions are also in progress. I’ll be sure to update y’all here when I have something new to announce.

[wz2b]

I checked @flaviu’s docs and looks good, I’m still a little scared of somehow bricking it.  I’m wondering, too, if you have a USB-to-serial dongle that controls RTS and DTR if you can have it control IO0 the usual way rather than having to manually ground it.  I’m nto even sure why this works (is there a pull-up on that port?)

The only thing that makes me a little asd is I’d like to have both … give me a nice clean ESPHOME interface (preferably via MQTT) but also still feed Emporia Cloud, since I do find that handy at times.

I’d also like to hear a few more people tell me “It worked! And nothing bad happened!”

[chrwei]

IO0 has to do with how ESP “programming mode” works.  it’s more than just the RTS pin, not hard, but not worth the effort for a one-off.  look up some of the ESP-01 programming adapters for examples

other than ESD damage, which shouldn’t be an issue if you ground yourself and Empoia did their engineering right, ESPs are pretty hard to brick.  there’s a basic protection circuits on each IO build into the model itself, and the bootloader is quite robust.

 

 

[magnetus]

can one replace the esp32 module and write the new code to the new module, i will like to keep the original module intact. I might just read and make a backup of it and then replace the module and install  the new code. also can we read the module if it has been updated? I have my vue2 connected for several months now and it has been updated

thanks

Miguel

[chrwei]

desoldering would be quite difficult, flaviu’s guide shows how to back up the stock firmware, restore is similar

[flaviu]

A bunch of great questions and concerns! I’ll keep these all in mind when I write an FAQ.

> I checked @flaviu’s docs and looks good, I’m still a little scared of somehow bricking it.  I’m wondering, too, if you have a USB-to-serial dongle that controls RTS and DTR if you can have it control IO0 the usual way rather than having to manually ground it.

I’ve never had a serial adaptor with RTS & DTR, but it looks like you can make it work! Connect EN & RTS, IO0 & DTR and you should be good to go.

> The only thing that makes me a little asd is I’d like to have both … give me a nice clean ESPHOME interface (preferably via MQTT) but also still feed Emporia Cloud, since I do find that handy at times.

MQTT support will likely happen in the near future (<6 mo).

Unfortunately, it is unlikely to ever be possible to use Emporia Cloud & ESPHome/Home Assistant simultaneously. It is technically possible, but would require reverse engineering the MQTT connection to AWS that Emporia Cloud uses, and would also likely make Emporia unhappy.

> can one replace the esp32 module and write the new code to the new module, i will like to keep the original module intact

I have quite advanced soldering skills, and I’d say this is too risky, even with my skills.

> I might just read and make a backup of it and then replace the module and install the new code.

Yup, you should absolutely do this. The instructions provide prominent guidance on how to do it, and are very insistent that you should not skip this step.

> can we read the module if it has been updated?

Should be no problem.

[err]

@flaviu wrote:

MQTT support will likely happen in the near future (<6 mo).

You mean an MQTT example right?

Because anyone can add an ESPHome MQTT client and using ESPHome automation to publish with no further work on your part.

https://esphome.io/components/mqtt.html

They just need to have fun with yaml syntax!

[flaviu]

> You mean an MQTT example right?

Just MQTT. At this time the Vue 2 integration uses a new framework called esp-idf due to a bug in the default arduino framework (which we’ve also resolved! but still need to test). The esp-idf feature in ESPHome is just two months old and does not yet support MQTT.

[mjankovits]

For anyone that has the Emporia-downloader program working, can you tell me what you used for the influx-token setting?  I’ve tried using the API token for my account, but I keep getting an “Invalide ID token” error.

23 Jan 2022 21:29:08 – ERROR (CognitoAuthenticationManager.java:123): Invalid ID token!
com.auth0.jwt.exceptions.InvalidClaimException: The Token can’t be used before Sun Jan 23 21:29:11 EST 2022.

I can get it to work with displaying the Vue data to the terminal window, but whenever I try to include my InfluxDB, it get the above error.

[wbarber69]

There is an integration for emporia through HACs. But it still requires cloud usage.  Which is slow and unreliable.  I do use it for automations though.  Stuff like the dishwasher or clothes dryer etc.  things that are t critical if they are a few minutes behind.

[Emperor XLII]

Just wanted to share my own PowerShell-based project (inspired largely by magico13’s PyEmVue): EmporiaRecorder

It is primarily focused on full data archiving for offline use, using a minimal number of API calls (different from real-time updates used for home automation or in the official app, though the underlying modules could certainly be used that way).  It still needs some polish to simplify setup/installation (which I sadly won’t be able to tackle in the near future), but any knowledgeable PowerShell user should be able to get it running (and happy to help with questions, when I am able).

[BFU_182461284]

Can someone elaborate on how to pull data from PyEmVue from kpnobvious post #7025 ? I got to that point and am completely lost. I read both the readme.md and API documentation but was unable to get past step  7.

• This reply was modified 2 years, 7 months ago by BFU_182461284.

[err]

ESPHome 2022.4.0 has been released with MQTT supporting using IDF. This means that Home Assistant is not strictly necessary if one deploys ESPHome + MQTT for the Vue2.

https://gist.github.com/flaviut/93a1212c7b165c7674693a45ad52c512#gistcomment-4142201

[toddhoustein]

Thanks for all of the work folk have done on various libraries! Total n00b question here… I’m using the PyEmVue python library. I’ve installed it and just trying to use the basic “Log in with username/password” from this page, but get an error from the very first line “NameError: name ‘PyEmVue’ is not defined”. I’m guessing I need to include a line that imports the library?

[toddhoustein]

So, I found the answer to my own question… You do indeed need an import statement, which (in case anyone else needs to know) is “from pyemvue import PyEmVue”

[akballow]

Dear Emporia,

What is you roadmap for the API. We have been noticing it has been neutered the past few weeks. Should we start looking for alternatives?

Thanks,
The Home Automation Scene

[wojo]

What has been neutered lately?

[chrwei]

the HASS integration is still working fine

[akballow]

Many Zeros in data now

[wojo]

I would open a ticket to investigate the issue, does the app also show the impact of 0s in the data when you use interface?

[akballow]

All my graphs and data have all these values from the API now. Notice where the line is thin. That is how it should be. App works fine funny enough its just the API which emporia is limiting.

[chrwei]

are you sure it’s not a network/internet issue? none of my graphs have zeros like that

[akballow]

Can you share your graphs?

[chrwei]

this one of my circuits, but i also tried replicating yours, and yeah, there’s something funcky on the panel

and unrelated, yours matching exactly with the solar is odd.

[akballow]

Oh my bar is energy at actually used, its a subtraction of solor and the at the meter, but yeah we have the same issue. More details here https://github.com/magico13/ha-emporia-vue/issues/137

• This reply was modified 2 years, 2 months ago by akballow.

[sarbuk]

I’m wondering if anyone can help me with how I find the authtoken I need to query the API directly at https://api.emporiaenergy.com ?

I’m not a developer so although I’ve looked through all the various github projects on integrating the Vue 2 with mostly Home Assistant, I can’t see anything obvious on how to get the authtoken that’s needed in the headers and deriving that from the web credentials.

I have a monitoring/graphing app I use (PRTG) that can query APIs directly and will parse the JSON response in whatever way I tell it to, I just need the API key/auth token.

[magico13]

It’s an Amazon Cognito OAuth token so the long way of getting the token from just a username and password is generally going to require some coding. The easier way is to let the web app do it for you by going to https://web.emporiaenergy.com and using your browser’s built in development tools to see the request/response to https://cognito-idp.us-east-2.amazonaws.com/ and pulling the tokens you need from the response data. The token is only valid for an hour so you’d need to call the refresh url every hour to get a new one, there’s sadly not a way to get a static API token for third party purposes like these (yet).

[sarbuk]

Ah, thanks @magico13, that’s really helpful! Looks like I need to utilise one of the methods kindly provided by others here.

[joe.king.cool]

@tedworksatemporia

first off, great product!!! i just checked the accuracy and its 99.98 percent accurate, probably more but electricity company rounds to nearest kwh. they came up with 1476 and after downloading csv i came up with 1473.4666.

and i fully understand why it don’t make since to have people accidently downloading years of data often, especially if there not paying a subscription. it costs way to much for the server load.

but, there’s always a but, lol…..why have this level of accuracy and all this data, but no useful or working filters in your app?? most of the filters from graphs didnt add up and if you scrolled to the timeframe you couldnt click on the bar to get the actual number.

basically if you just had one filter on your app.

1 start time/date

2 end time/date

3 the device in question ex: frig ex: main etc

4 the unit ex: dollars  ex: kwh etc

those 4 questions then give the answer. then allot of these people wouldn’t needed to go threw all this work to get data then spend time manipulating it and overloading your servers.

if you want examples of what dont make sense. i can give to you. or you could just add a calculator section with those 4 question, and give them the answer.

understand some folks want to see trends for each circuit as well. you could also put a filter in the for how much each cuircut did each month, based off the bill cycle of electricity company.

and if your looking for a way to generate more income for your company. i suggest charge 5 per month, low enough you shouldn’t get any complaints, for the extra detail features. then sweeten the deal with extended warranty for devices as long as you have the subscription for a certain amount of time. now you have reoccurring income from past customers to pay for the server load and future tweaks to the app. have a free version app which is the current way and a paid version that unlocks the features.

[rowan.bradley]

What I need to do is to calculate whether adding battery storage to my PV system will over time save me money. To do this I seem to need to know:

1. How much energy am I sending back to the grid when the PV is generating more that I am using. I need this every short period (a few minutes?) for an extended period, long enough to be representative.
2. How much energy am I using when the PV is no0t generating enough? How much of this is during peak times, and how much off peak? I need this too over an extended period of time.
3. If I know the capacity, the peak output current, the efficiency etc. of the battery system, and the price of electricity during peak and off peak times, then I think I could calculate how much the battery system would save me, and thence the payback period.

I have an Emporia Vue monitoring the output of my PV system, the total energy draw from or fed back to the grid, as well as a number of individual circuits in my home. What is the best way of getting the data that I need and doing this calculation? Can I use helgew’s emporia-downloader? Or is there a better way?

Thank you – Rowan

<script src=”chrome-extension://fbnkflkahhlmhdgkddaafgnnokifobik/assets/pageScripts.js”></script>

[sarbuk]

If this is just one time that you want to export this data, then you can do that using Emporia’s built-in export tool. In the app, go to the menu in the top left, scroll to “Export Raw Data to CSV”, and then choose your device, and then dates. The data will be made available by a link that’s emailed to you. It will be a zip file containing CSV files for per second, per minute, per hour and per day data. Possibly per month as well, I can’t remember.

That should give you everything you need to calculate if a battery is of any use to you.

[techwizard]

Greetings,

Rest API is now available!!!

I’ve dedicated considerable effort to crafting a user-friendly REST API. I warmly encourage you to visit https://emporia-connect.xyt.co.za and share your insights, suggestions, or any hurdles you encounter. While this version is an early iteration, I’m devoted to its ongoing refinement. Your valuable feedback is immensely appreciated!

Moreover, I’d like to emphasize that this API is entirely free.

You can contact me at emporia-connect@xyt.co.za

• This reply was modified 11 months, 2 weeks ago by techwizard.

[sarbuk]

This looks really interesting, but I have a few questions.

1. What’s the flow of data? Is this API an intermediary between the Emporia servers and the API endpoints?
2. If so, how are you protecting data in transit?
3. Are you storing any data, and if so, how is this protected?
4. How are you protecting credentials and what risk is there to me submitting my credentials on your platform?
5. How are you planning to cover costs when this scales up?

You could do with including some basic instructions on how to get started. I’m making some guesses, but I’m assuming I need to click “Authorize” first, enter my Emporia creds, then use Postman or some other API tool to run queries using a token and URL that is subsequently given, but that’s not clear.

I do appreciate that you’ve put effort into this, so thank you!

 

[techwizard]

Hi Sarbuk,

Thank you for taking the time to explore api. I’m thrilled to hear about your interest and your questions are absolutely valid and important to address. Let me provide you with some insights:

1. Flow of Data: Yes, the API acts as an intermediary between Emporia servers and the API endpoints. It facilitates the seamless transfer of data.
2. Data Protection during Transit: We prioritize data security during transit through robust encryption protocols, ensuring that data remains protected while in motion.
3. Data Storage and Protection: It’s important to note that we do not store any user data. Our system is designed to operate without the need for data storage, ensuring your information remains solely within the designated endpoints.
4. Credential Protection: Safeguarding credentials is of paramount importance to us. We employ stringent measures to encrypt and protect your credentials. Your trust and security are our top priorities.
5. Scaling and Cost Management: At present, our application is hosted in a specific environment. However, we are in the process of transitioning to Docker, which offers enhanced scalability through cluster capabilities. This move will empower us to efficiently manage increasing demands without compromising performance or incurring exorbitant costs. We’re actively working on this transition to ensure a seamless and scalable infrastructure as we grow.
6. Getting Started Instructions: To start using the API, you can utilize basic authentication with each request. An example using cURL is provided below
   curl -X GET -u username:password https://emporia-connect.xyt.co.za/api
   Replace username(email address) and password with your actual Emporia credentials. This cURL command demonstrates how to perform a GET request to the specified endpoint (https://emporia-connect.xyt.co.za/api) using basic authentication.

Your feedback and inquiries are incredibly valuable in aiding us to improve user experience. Should you have further questions or require assistance, please feel free to reach out.

[chrwei]

I have the same questions as sarbuk, though since I’ve done extensive API usage and creation the basic usage is actually quite clear to me.  I can see less experienced people needing a guide though.

What i do find missing is examples of what gets returned.  You have nothing at all, requiring experimentation to discover it.  Examples would let me decide if this even gives what I’d need before sending my creds through your server, which as already stated is a concern.

I’m also interested to hear what you think the use case is for this.  For developers, there’s several local options on github already, many of which can easily host a local rest api.  for example, HomeAssistant has a robust rest API that can pull data from any integration with authorization tokens, which is safer than base64 encoded creds for less secure client tools.

[techwizard]

Hi Chrwei,

Thank you for taking the time to explore api.

Let me provide you with some insights:

1. Flow of Data: Yes, the API acts as an intermediary between Emporia servers and the API endpoints. It facilitates the seamless transfer of data.
2. Data Protection during Transit: We prioritize data security during transit through robust encryption protocols, ensuring that data remains protected while in motion.
3. Data Storage and Protection: It’s important to note that we do not store any user data. Our system is designed to operate without the need for data storage, ensuring your information remains solely within the designated endpoints.
4. Credential Protection: Safeguarding credentials is of paramount importance to us. We employ stringent measures to encrypt and protect your credentials. Your trust and security are our top priorities.
5. Scaling and Cost Management: At present, our application is hosted in a specific environment. However, we are in the process of transitioning to Docker, which offers enhanced scalability through cluster capabilities. This move will empower us to efficiently manage increasing demands without compromising performance or incurring exorbitant costs. We’re actively working on this transition to ensure a seamless and scalable infrastructure as we grow.
6. Getting Started Instructions: To start using the API, you can utilize basic authentication with each request. An example using cURL is provided below
   curl -X GET -u username:password https://emporia-connect.xyt.co.za/api
   Replace username(email address) and password with your actual Emporia credentials. This cURL command demonstrates how to perform a GET request to the specified endpoint (https://emporia-connect.xyt.co.za/api) using basic authentication.

Additionally, we’ve made recent updates to our website and documentation to enhance user experience and clarity. Please feel free to review these updates along with the API. Your feedback is invaluable in helping us refine our services and provide a better experience for our users.

Your questions, suggestions, or further inquiries are always welcomed. Don’t hesitate to reach out if you need any assistance.

[sarbuk]

Hi chrwei

I’m most interested in this for pulling live usage data into HomeAssistant for use in automation, and into a monitoring platform for record keeping. I am using a HomeAssistant integration that doesn’t pull through live data but it does pull through hourly historic data. Are there any integrations currently available to pull live data out of Emporia?

The risk to credentials with techwizard’s API is relatively low as long as you’re using unique credentials for Emporia (which everyone should be these days), and Emporia is a simple read-only system, so there’s not much that could go wrong if that data got out or control was gained over my Emporia account (although that is different for customers using their smart plugs and/or EV charger, possibly). But even so, I want to know more before I commit!

Thanks!

 

[chrwei]

https://github.com/magico13/ha-emporia-vue pulls down to 1 minute, which i find plenty for automations.  it does have some issues, requiring a reload to work again, I’ve posted an automation that takes care of that too https://github.com/magico13/ha-emporia-vue/issues/224#issuecomment-1803935930

your creds can be used to submit data, as well as get some demographic info, still, you’re right that it’s low risk, but it’s still a concern.  The site simply lacks the warm-fuzzies of some statement about data use and privacy.

[techwizard]

We are excited to announce recent updates to the Emporia Connect API, aimed at providing a more robust and user-friendly experience. Your input and feedback are invaluable as we strive to enhance our services continuously.

New Features:

1. Improved Security Measures: We’ve bolstered our security protocols to ensure your data remains safe and protected.
2. Enhanced Documentation: Updated documentation for easier navigation and clearer instructions on utilizing the API.
3. Streamlined Website: Our website has been revamped to offer a more intuitive and informative browsing experience.

To explore these updates, we invite you to review the enhanced Emporia Connect API and associated documentation on our website (emporia-connect.xyt.co.za). Your insights into the changes made will greatly assist us in refining our services.

Your participation in this process is invaluable to us. Please take a moment to browse through the updated features and share your thoughts, suggestions, or inquiries you may have.

[chrwei]

nice improvements!

[Author]

Posts