Discussion:
[Nut-upsdev] Patching CyberPower UPS drivers
Ben Kamen
2016-01-19 14:37:29 UTC
Permalink
Hey there,

I got a new UPS (CyberPower PR1500LCDRTXL2Ua) and found the serial drivers don't quite work...

So I decided to jump in and make changes to the driver that would hopefully make it happier.

I have it working (I really only changed powerp-txt.c) but still have a couple questions... Charles said I should mention them here to talk to anyone interest but specifically Arjen (one of the original authors of the driver)

Soooo...

Here I am. ;)


Where should we start?

Essentially, I rewrote powpan_status so it better tolerates strings of different lengths (although I only have the 1 UPS right now, so I can't test against another and different string lengths)

I had to make the serial read much longer so it grabs 55bytes instead of the original short list.
9.843588 read: (55 bytes) => 23 49 31 31 38 2e 30 4f 31 31 38 2e 30 4c 30 31 37 42
9.843663 31 30 30 54 30 32 35 48 30 36 30 2e 30 46 30 36 30 2e 30 52 30 35 31 51 30
9.843695 30 32 53 90 84 c0 88 80 57 00 81 0d
compared to the original string support in the file, there's now 2 fields of binary data (in ASCII Protocol? seriously? smh)

and the S field on this UPS has 4 bytes instead of just 2.

(not that it matters I guess... but now it makes me wonder what all the extra information is)

(I also guess someone has tried reaching out to CyberPower for docs and been turned down?)


Anyway -- in rewriting this, the only part I have left to do to where I'm happy is setting the timeout.

Instead of asking for N number of chars and then timing out after a long time, I'd like to ask it for a string (up to the 0x0D).

I haven't written that in yet, but will if someone says there's another way -- or I have to do it that way....

so that an incomplete string after time causes a timeout but a string that comes in reasonably soon will be collected -- but not using a fixed number (since it seems these UPSs now have variability in string length between models) and then just parse the string.

The parsing part works. I just need to alter the collection part so the timeouts work differently.

Any advice from anyone?


Thanks,

-Ben
Charles Lepple
2016-01-28 14:18:39 UTC
Permalink
This post might be inappropriate. Click to display it.
Ben Kamen
2016-01-28 16:57:01 UTC
Permalink
I got 2 copies of that email.

weird. Was it just me?

Thanks for the look and return diff.

I'll take a look.

-Ben
Ben Kamen
2016-01-28 17:03:22 UTC
Permalink
Post by Charles Lepple
I definitely like your approach. Thanks for moving some of the "magic numbers" to named constants, as well.
Thanks. Always nice to have real coders sanity check anything I've written with a "thumbs up" response. (I'm more a hardware guy)

And sscanf was working like poo -- so I figured I'd parse it differently with something that didn't seem so sensitive.
Post by Charles Lepple
* Since v2.6.5, some of the string functions were renamed. rtrim() -> str_rtrim(), for instance.
I'll take a look at the latest.... as mentioned, 2.6.5 is what's on my CentOS system with nothing later (yet) so I just downloaded the SRPMS to work on.

but I also have 2.7.3 downloaded -- so I'll see what I can do with that.
Post by Charles Lepple
* If you are taking out a block of code, "#if 0" is better than comment characters (which don't nest), and removing it completely is even better (fewer false positives when searching). If nothing else, it's in version control.
Yea - the comments were just temporary to keep the old stuff in my view while I was re-writing around it.

Then once it worked and I planned on removing it.

I'll send along something more finished once I'm done.

Thanks again!

-Ben
Charles Lepple
2016-01-29 04:26:49 UTC
Permalink
Post by Ben Kamen
I'll take a look at the latest.... as mentioned, 2.6.5 is what's on my CentOS system with nothing later (yet) so I just downloaded the SRPMS to work on.
but I also have 2.7.3 downloaded -- so I'll see what I can do with that.
Well, as long as we know the version that you are starting from, it's not too hard to merge.

But if you're downloading later versions, I'd like to gently point you in the direction of the Git tree, or the tarball snapshots from Buildbot:

http://www.networkupstools.org/docs/developer-guide.chunked/ar01s03.html#_source_code_management

http://buildbot.networkupstools.org/snapshots (note that "2.7.3.1" is not a real version number, it's just something greater than 2.7.3 - the links include the full Git revision, and only the first 5-8 characters are needed from that)
--
Charles Lepple
***@gmail
Ben Kamen
2017-06-10 19:06:42 UTC
Permalink
Hey all,

I recently got an RMCARD205 for my Cyber UPS....
Network UPS Tools - UPS driver controller 2.6.5
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
No matching MIB found for sysOID '.1.3.6.1.4.1.3808.1.1.1'!
Please report it to NUT developers, with an 'upsc' output for your device.
Going back to the classic MIB detection method.
Detected PR1500LCDRTXL2Ua on host 192.168.125.10 (mib: cyberpower 0.1)
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected SMART-UPS 3000 RM on host 192.168.125.7 (mib: apcc 1.2)
[shop] Warning: excessive poll failures, limiting error reporting
[shop] Warning: excessive poll failures, limiting error reporting
If I use the mibs = cyberpower
Network UPS Tools - UPS driver controller 2.6.5
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected PR1500LCDRTXL2Ua on host 192.168.125.10 (mib: cyberpower 0.1)
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected SMART-UPS 3000 RM on host 192.168.125.7 (mib: apcc 1.2)
[shop] Warning: excessive poll failures, limiting error reporting
[shop] Warning: excessive poll failures, limiting error reporting
but could use updating as the upsc output is kinda lacking some desired measurements (that I get with APC) ... I see the current is 2.7.x but I'm on CentOS/RH 6.x which seems to have stalled out at 2.6.5

Does 2.7 have the newer fancy MIB and I can just compile up and install manually?

I'm assuming RH/CentOS 6 won't be getting anymore updates. :(

I'll look around some more for an RPM for 6.x

Cheers,

-Ben


Also,

when I go to : http://networkupstools.org/docs/man/snmp-ups.html and click on the link at the bottom: http://www.networkupstools.org/protocols/snmp/ I get a 404 error. (just an FYI)
Tim Dawson
2017-06-10 20:26:47 UTC
Permalink
Build from source, and done . . . wasting your time looking for a less out of date RPM is pointless. . . RPM, .deb, etc are almost always downrev. . .

- Tim
Post by Ben Kamen
Hey all,
I recently got an RMCARD205 for my Cyber UPS....
Network UPS Tools - UPS driver controller 2.6.5
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
No matching MIB found for sysOID '.1.3.6.1.4.1.3808.1.1.1'!
Please report it to NUT developers, with an 'upsc' output for your
device.
Going back to the classic MIB detection method.
Detected PR1500LCDRTXL2Ua on host 192.168.125.10 (mib: cyberpower
0.1)
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected SMART-UPS 3000 RM on host 192.168.125.7 (mib: apcc 1.2)
[shop] Warning: excessive poll failures, limiting error reporting
[shop] Warning: excessive poll failures, limiting error reporting
If I use the mibs = cyberpower
Network UPS Tools - UPS driver controller 2.6.5
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected PR1500LCDRTXL2Ua on host 192.168.125.10 (mib: cyberpower
0.1)
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected SMART-UPS 3000 RM on host 192.168.125.7 (mib: apcc 1.2)
[shop] Warning: excessive poll failures, limiting error reporting
[shop] Warning: excessive poll failures, limiting error reporting
but could use updating as the upsc output is kinda lacking some desired
measurements (that I get with APC) ... I see the current is 2.7.x but
I'm on CentOS/RH 6.x which seems to have stalled out at 2.6.5
Does 2.7 have the newer fancy MIB and I can just compile up and install manually?
I'm assuming RH/CentOS 6 won't be getting anymore updates. :(
I'll look around some more for an RPM for 6.x
Cheers,
-Ben
Also,
when I go to : http://networkupstools.org/docs/man/snmp-ups.html and
http://www.networkupstools.org/protocols/snmp/ I get a 404 error. (just
an FYI)
_______________________________________________
Nut-upsdev mailing list
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/nut-upsdev
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Ben Kamen
2017-06-11 03:24:04 UTC
Permalink
Post by Tim Dawson
Build from source, and done . . . wasting your time looking for a less out of date RPM is pointless. . . RPM, .deb, etc are almost always downrev. . .
True...

And I already built the source - just have to install.

However, before I go through this -- how to I check which SNMP mibs are in use for a given net management card (like the RMCARD205)

I downloaded the latest MIB from CyberPower -- but it looks like "so?" -- that doesn't help me?

I'm looking at snmp-ups.c but could use little guidance for how the SNMP part of this works.


-Ben
Manuel Wolfshant
2017-06-11 06:58:11 UTC
Permalink
Post by Tim Dawson
Build from source, and done . . . wasting your time looking for a less
out of date RPM is pointless. . . RPM, .deb, etc are almost always
downrev. . .
- Tim
Post by Ben Kamen
Hey all,
I recently got an RMCARD205 for my Cyber UPS....
Network UPS Tools - UPS driver controller 2.6.5
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
No matching MIB found for sysOID '.1.3.6.1.4.1.3808.1.1.1'!
Please report it to NUT developers, with an 'upsc' output for your
device.
Going back to the classic MIB detection method.
Detected PR1500LCDRTXL2Ua on host 192.168.125.10 (mib: cyberpower
0.1)
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected SMART-UPS 3000 RM on host 192.168.125.7 (mib: apcc 1.2)
[shop] Warning: excessive poll failures, limiting error reporting
[shop] Warning: excessive poll failures, limiting error reporting
If I use the mibs = cyberpower
Network UPS Tools - UPS driver controller 2.6.5
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected PR1500LCDRTXL2Ua on host 192.168.125.10 (mib: cyberpower
0.1)
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected SMART-UPS 3000 RM on host 192.168.125.7 (mib: apcc 1.2)
[shop] Warning: excessive poll failures, limiting error reporting
[shop] Warning: excessive poll failures, limiting error reporting
but could use updating as the upsc output is kinda lacking some
desired
Post by Ben Kamen
measurements (that I get with APC) ... I see the current is 2.7.x but
I'm on CentOS/RH 6.x which seems to have stalled out at 2.6.5
Does 2.7 have the newer fancy MIB and I can just compile up and
install
Post by Ben Kamen
manually?
I'm assuming RH/CentOS 6 won't be getting anymore updates. :(
I'll look around some more for an RPM for 6.x
Cheers,
-Ben
Also,
when I go to : http://networkupstools.org/docs/man/snmp-ups.html and
http://www.networkupstools.org/protocols/snmp/ I get a 404 error.
(just
Post by Ben Kamen
an FYI)
_______________________________________________
Nut-upsdev mailing list
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/nut-upsdev
I can upload to http://wolfy
.fedorapeople
org the packages I built for 2.7.4, if you are willing to test them. I am using them for several months.

Wolfy
Jim Klimov
2017-06-11 12:22:38 UTC
Permalink
Post by Manuel Wolfshant
Post by Tim Dawson
Build from source, and done . . . wasting your time looking for a less
out of date RPM is pointless. . . RPM, .deb, etc are almost always
downrev. . .
- Tim
Post by Ben Kamen
Hey all,
I recently got an RMCARD205 for my Cyber UPS....
Network UPS Tools - UPS driver controller 2.6.5
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
No matching MIB found for sysOID '.1.3.6.1.4.1.3808.1.1.1'!
Please report it to NUT developers, with an 'upsc' output for your
device.
Going back to the classic MIB detection method.
Detected PR1500LCDRTXL2Ua on host 192.168.125.10 (mib: cyberpower
0.1)
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected SMART-UPS 3000 RM on host 192.168.125.7 (mib: apcc 1.2)
[shop] Warning: excessive poll failures, limiting error reporting
[shop] Warning: excessive poll failures, limiting error reporting
If I use the mibs = cyberpower
Network UPS Tools - UPS driver controller 2.6.5
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected PR1500LCDRTXL2Ua on host 192.168.125.10 (mib: cyberpower
0.1)
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected SMART-UPS 3000 RM on host 192.168.125.7 (mib: apcc 1.2)
[shop] Warning: excessive poll failures, limiting error reporting
[shop] Warning: excessive poll failures, limiting error reporting
but could use updating as the upsc output is kinda lacking some
desired
Post by Ben Kamen
measurements (that I get with APC) ... I see the current is 2.7.x but
I'm on CentOS/RH 6.x which seems to have stalled out at 2.6.5
Does 2.7 have the newer fancy MIB and I can just compile up and
install
Post by Ben Kamen
manually?
I'm assuming RH/CentOS 6 won't be getting anymore updates. :(
I'll look around some more for an RPM for 6.x
Cheers,
-Ben
Also,
when I go to : http://networkupstools.org/docs/man/snmp-ups.html and
http://www.networkupstools.org/protocols/snmp/ I get a 404 error.
(just
Post by Ben Kamen
an FYI)
_______________________________________________
Nut-upsdev mailing list
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/nut-upsdev
I can upload to http://wolfy
.fedorapeople
org the packages I built for 2.7.4, if you are willing to test them.
I am using them for several months.
Wolfy
_______________________________________________
Nut-upsdev mailing list
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/nut-upsdev
Hi.

The SNMP support in NUT works by having a large number of mapping tables that correlate certain OIDs (exact or recently patterned, for arrays of similar data points) to NUT keys, and sometimes define conversion functions to get standard-units values. The MIB files are not directly used for this at runtime, though are often invaluable to create or maintain these mapping tables as hardware support evolves.

UPS management cards might refer to their standard entry point, which allows the driver to guess a relevant MIB as recommended by vendor, though there are cases when another MIB matches better in practice - so there is an option to pick it explicitly. NUT can also probe these entry points that it knows from mapping data, to see if it gets a meaningful response. Also most devices provide some data (without fancy vendor-specific features) in the IETF standard MIB subtree, so it is the last fallback choice in automatic guesswork.
Post by Manuel Wolfshant
Does 2.7 have the newer fancy MIB and I can just compile up and
install manually?
I cherish this part of your question ;) as I spent a large part of last year co-developing the DMF (dynamic mapping format for me, though there are other de-abbreviations) technology which allows just that - give the capable version of snmp-ups the suitable mapping file, reload the driver and try out support for new hardware with unchanged binaries. This mapping file is an XML with mappings described above - whereas the original NUT mappings are in C code and stuff must be recompiled to change this data.

Currently the DMF branch is in PR limbo (being a large chunk of added code), and there were questions whether anyone needs it beside our forked project this was made for. IMHO it would be useful in cases like yours as well, where people are stuck with whatever their distro offers (perhaps due to compliance and certification, or lack of time or skill to re-roll their own package), so please speak up to sway the balance ;)

Finally, as you would compile your own, consider taking the github upstream/master branch - it has quite advanced ahead of the latest official tarballed release.

Jim

--
Typos courtesy of K-9 Mail on my Redmi Android
Ben
2017-06-19 04:53:09 UTC
Permalink
Post by Jim Klimov
Post by Manuel Wolfshant
Post by Tim Dawson
Build from source, and done . . . wasting your time looking for a less
out of date RPM is pointless. . . RPM, .deb, etc are almost always
downrev. . .
- Tim
Post by Ben Kamen
Hey all,
I recently got an RMCARD205 for my Cyber UPS....
Network UPS Tools - UPS driver controller 2.6.5
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
No matching MIB found for sysOID '.1.3.6.1.4.1.3808.1.1.1'!
Please report it to NUT developers, with an 'upsc' output for your
device.
Going back to the classic MIB detection method.
Detected PR1500LCDRTXL2Ua on host 192.168.125.10 (mib: cyberpower
0.1)
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected SMART-UPS 3000 RM on host 192.168.125.7 (mib: apcc 1.2)
[shop] Warning: excessive poll failures, limiting error reporting
[shop] Warning: excessive poll failures, limiting error reporting
If I use the mibs = cyberpower
Network UPS Tools - UPS driver controller 2.6.5
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected PR1500LCDRTXL2Ua on host 192.168.125.10 (mib: cyberpower
0.1)
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected SMART-UPS 3000 RM on host 192.168.125.7 (mib: apcc 1.2)
[shop] Warning: excessive poll failures, limiting error reporting
[shop] Warning: excessive poll failures, limiting error reporting
but could use updating as the upsc output is kinda lacking some
desired
Post by Ben Kamen
measurements (that I get with APC) ... I see the current is 2.7.x but
I'm on CentOS/RH 6.x which seems to have stalled out at 2.6.5
Does 2.7 have the newer fancy MIB and I can just compile up and
install
Post by Ben Kamen
manually?
I'm assuming RH/CentOS 6 won't be getting anymore updates. :(
I'll look around some more for an RPM for 6.x
Cheers,
-Ben
Also,
when I go to : http://networkupstools.org/docs/man/snmp-ups.html and
http://www.networkupstools.org/protocols/snmp/ I get a 404 error.
(just
Post by Ben Kamen
an FYI)
_______________________________________________
Nut-upsdev mailing list
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/nut-upsdev
I can upload to http://wolfy
.fedorapeople
org the packages I built for 2.7.4, if you are willing to test them.
I am using them for several months.
Wolfy
_______________________________________________
Nut-upsdev mailing list
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/nut-upsdev
Hi.
The SNMP support in NUT works by having a large number of mapping tables that correlate certain OIDs (exact or recently patterned, for arrays of similar data points) to NUT keys, and sometimes define conversion functions to get standard-units values. The MIB files are not directly used for this at runtime, though are often invaluable to create or maintain these mapping tables as hardware support evolves.
UPS management cards might refer to their standard entry point, which allows the driver to guess a relevant MIB as recommended by vendor, though there are cases when another MIB matches better in practice - so there is an option to pick it explicitly. NUT can also probe these entry points that it knows from mapping data, to see if it gets a meaningful response. Also most devices provide some data (without fancy vendor-specific features) in the IETF standard MIB subtree, so it is the last fallback choice in automatic guesswork.
I guessed some of that from seeing out some of the H files were set up (without digging too far)....
Post by Jim Klimov
Post by Manuel Wolfshant
Does 2.7 have the newer fancy MIB and I can just compile up and install manually?
I cherish this part of your question ;) as I spent a large part of last year co-developing the DMF (dynamic mapping format for me, though there are other de-abbreviations) technology which allows just that - give the capable version of snmp-ups the suitable mapping file, reload the driver and try out support for new hardware with unchanged binaries. This mapping file is an XML with mappings described above - whereas the original NUT mappings are in C code and stuff must be recompiled to change this data.
Currently the DMF branch is in PR limbo (being a large chunk of added code), and there were questions whether anyone needs it beside our forked project this was made for. IMHO it would be useful in cases like yours as well, where people are stuck with whatever their distro offers (perhaps due to compliance and certification, or lack of time or skill to re-roll their own package), so please speak up to sway the balance ;)
Finally, as you would compile your own, consider taking the github upstream/master branch - it has quite advanced ahead of the latest official tarballed release.
Thanks for the elaborate answer here.

I suppose my inquiry could speak for "speaking up" as it would have been very handy to roll a custom mapping file.

I already downloaded the MIB from CyberPower and can see it should supply all the things I'm spoiled with using APC. (except they overcharge their batteries. Don't get me started. ;) )

I'll take a look at the master branch and holla if I get stuck.

Thanks again,

-Ben
Carsten Burkhardt
2017-09-21 13:33:18 UTC
Permalink
Hi Tim.
Post by Tim Dawson
looking for a less out of date RPM
Please have a look on following links:

http://metadata.ftp-master.debian.org/changelogs/main/n/nut/nut_2.7.4-5_changelog
http://networkupstools.org/source/2.7/new-2.7.4.txt
The version look similar and actually.
Post by Tim Dawson
.deb, etc are almost always downrev. . .
That is not true. The maintainer make a quite pretty work. I have to thank for always new version in Debian.
Post by Tim Dawson
Build from source, and done . . . wasting your time looking for a less out of date RPM is pointless. . . RPM, .deb, etc are almost always downrev. . .
- Tim
Hey all,
I recently got an RMCARD205 for my Cyber UPS....
Network UPS Tools - UPS driver controller 2.6.5
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
No matching MIB found for sysOID '.1.3.6.1 <http://1.3.6.1>.4.1.3808.1.1.1 <http://8.1.1.1>'!
Please report it to NUT developers, with an 'upsc' output for your device.
Going back to the classic MIB detection method.
Detected PR1500LCDRTXL2Ua on host 192.168.125.10 <http://192.168.125.10> (mib: cyberpower 0.1)
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected SMART-UPS 3000 RM on host 192.168.125.7 <http://192.168.125.7> (mib: apcc 1.2)
[shop] Warning: excessive poll failures, limiting error reporting
[shop] Warning: excessive poll failures, limiting error reporting
If I use the mibs = cyberpower
Network UPS Tools - UPS driver controller 2.6.5
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected PR1500LCDRTXL2Ua on host 192.168.125.10 <http://192.168.125.10> (mib: cyberpower 0.1)
Network UPS Tools - Generic SNMP UPS driver 0.68 (2.6.5)
Detected SMART-UPS 3000 RM on host 192.168.125.7 <http://192.168.125.7> (mib: apcc 1.2)
[shop] Warning: excessive poll failures, limiting error reporting
[shop] Warning: excessive poll failures, limiting error reporting
but could use updating as the upsc output is kinda lacking some desired measurements (that I get with APC) ... I see the current is 2.7.x but I'm on CentOS/RH 6.x which seems to have stalled out at 2.6.5
Does 2.7 have the newer fancy MIB and I can just compile up and install manually?
I'm assuming RH/CentOS 6 won't be getting anymore updates. :(
I'll look around some more for an RPM for 6.x
Cheers,
-Ben
Also,
when I go to :http://networkupstools.org/docs/man/snmp-ups.html and click on the link at the bottom:http://www.networkupstools.org/protocols/snmp/ I get a 404 error. (just an FYI)
Mit freundlichen Grüßen

Carsten Burkhardt
--
Dipl. Ing.Carsten Burkhardt | ***@b-c-s.de
bcs kommunikationslösungen
Inh. Dipl. Ing. Carsten Burkhardt
Harz 51 | 06108 Halle (Saale) | Germany
tel +49 345 29849-0 | fax +49 345 29849-22
www.b-c-s.de | www.halle.it | www.wivewa.de

Unsere 4. Veranstaltung zum Thema "LINUX FÜR (H)ALLE" war dank der Besucher und Helfer wieder ein Erfolg.
http://www.halle.it/linux-presentation-day/

SIE MÖCHTEN EINFACH IHRE ADRESSEN, GEBURTSTAGE UND DOKUMENTE VERWALTEN UND TROTZDEM IHRE DATEN SICHER WISSEN. DANN ÜBERZEUGEN SIE SICH VON WIVEWA unter: www.wivewa.de
Loading...