tobru
/
pylokid
1
0
Fork 0
Code Pull Requests Releases Activity

Merge pull request #1 from tobru/mechsoup 

refactoring again
This commit is contained in:
Tobias Brunner 2018-01-02 18:41:23 +01:00 committed by GitHub
parent 461638d13f 4b3a1456e1
commit b2c736ec5c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
10 changed files with 507 additions and 301 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

131
README.md
View File

@ -2,50 +2,129 @@
"Loki - Nordischer Gott des Feuers"
This app helps a Feuerwehr Fourier from the Canton of Zurich
in Switzerland to ease the pain of the huge work for getting
Einsätze correctly into [Lodur](https://www.lodur.ch/lodur.html).
*General application description is in German*
## Idea
**pylokid** ist eine Hilfsapplikation um z.B. Einsatzaufträge, welche die
**Einsatzleitzentrale (ELZ)** per E-Mail versendet, automatisch im
**[Lodur](https://www.lodur.ch/lodur.html)** einzutragen.
Die Applikation versucht so viele Informationen über den Einsatz in
Erfahrung zu bringen, wie nur möglich. Dies geschieht unter anderem
durch Auslesen von Daten aus dem der E-Mail angehängten PDF.
* Get mails sent by ELZ with subjects
"Einsatzausdruck_FW" and "Einsatzprotokoll"
* Store attached PDF in Feuerwehr Cloud (WebDAV)
* Publish new message over MQTT
* Parse PDFs and try to get information about Einsatz
* Connect to Lodur and create a new Einsatzrapport with
as much information as possible
Im Moment ist die Applikation vermutlich nur im **Kanton Zürich** einsetzbar
und vermutlich auch nur für die **[Feuerwehr Urdorf](https://www.feuerwehrurdorf.ch/)**.
Bei Interesse an dieser Applikation von anderen Feuerwehren bin ich
gerne bereit, diese entsprechend zu generalisieren und
[weiter zu entwickeln](https://github.com/tobru/pylokid/issues/new).
## Funktionsweise
Bei einem Feuerwehralarm sendet die ELZ automatisch eine E-Mail
mit einem PDF im Anhang, welches alle notwendigen Informationen
zum Einsatz enthält. Nach Abschluss des Einsatzes sendet die ELZ
ein weiteres E-Mail mit dem Einsatzprotokoll.
pylokid funktioniert so:
* Alle x Sekunden wird das angegebene Postfach nach ELZ E-Mails
überprüft. Diese identifizieren sich mit dem Betreff
"Einsatzausdruck_FW" oder "Einsatzprotokoll".
* Wird ein passendes E-Mail gefunden, wird der Anhang (das PDF)
heruntergeladen, in die Cloud gespeichert (WebDAV) und im Lodur
ein entsprechender Einsatzrapport eröffnet und vorausgefüllt.
Das PDF wird sinnvoll umbenannt und als Alarmdepesche ins Lodur
geladen.
* Kommen weitere E-Mails mit dem Betreff "Einsatzausdruck_FW" werden
diese im Lodur am entsprechenden Einsatzrapport angehängt.
* Ist der Einsatz abgeschlossen und das Einsatzprotokoll eingetroffen
werden die Informationen im Lodur nachgetragen.
Desweiteren wird über MQTT eine Nachricht mit möglichst vielen
Informationen publiziert, inkl. dem PDF. So kann ein Drittsystem
auf einen Einsatz reagieren, z.B. den Einsatz auf einem Dashboard
darstellen.
## Stolpersteine / Herausforderungen
Im abgebildeten Prozess gibt es viele Stolpersteine. Zwei davon:
* Die ELZ sendet PDFs welche by design keine Datenstruktur haben.
Somit ist das herausholen von Informationen mehr Glückssache
als Verstand. Würde die ELZ die Informationen vom PDF als
strukturierte Daten liefern, würde das System um einiges stabiler.
* Lodur hat keine API. Alle Datenmanipulationen funktioniert über
Reverse Engineering der HTML Formulare. Dabei kamen einige
spezielle Techniken von Lodur zum Vorschein:
* Nach dem Anlegen eines Einsatzrapportes wird in der Antwort des
Servers dieselbe Seite mit einem JavaScript Script gesendet,
welches die Browserseite noch einmal neu lädt und zum angelegten
Datensatz weiterleitet. Nur in diesem JavaScript Tag findet man
die zugewiesene Datensatz ID.
* Zur Bearbeitung eines Datensatzes werden die bekannten Daten nicht
etwa im HTML Formular als "value" eingetragen, sondern via
JavaScript ausgefüllt. Und es müssen immer alle Daten nochmals
gesendet werden, inkl. einiger hidden Fields.
Um die Probleme mit Lodur zu umgehen, werden alle Daten, welche
an Lodur gesendet werden, in einem JSON File im WebDAV neben den
PDFs abgelegt. So lässt sich im Nachhinein ein Datensatz bearbeiten
und eine Zuordnung des Einsatzes im WebDAV und in Lodur herstellen.
## Installation and Configuration
The application is written in Python and runs perfectly in OpenShift
with the Python S2I builder.
Configuration is done via environment variables:
* *IMAP_SERVER*: Adress of IMAP server
* *IMAP_USERNAME*: Username for IMAP login
* *IMAP_PASSWORD*: Password of IMAP user
* *IMAP_MAILBOX*: IMAP Mailbox to check for matching messages. Default: INBOX
* *IMAP_CHECK_INTERVAL*: Interval in seconds to check mailbox. Default: 30
* *WEBDAV_URL*: Complete WebDAV URL
* *WEBDAV_USERNAME*: Username for WebDAV
* *WEBDAV_PASSWORD*: Password of WebDAV user
* *WEBDAV_BASEDIR*: Basedir on WebDAV
* *TMP_DIR*: Temporary directory. Default: /tmp
* *MQTT_SERVER*: Address of MQTT Broker
* *MQTT_USER*: Username for MQTT login
* *MQTT_PASSWORD*: Password of MQTT user
* *MQTT_BASE_TOPIC*: Base topic to publish information
* *LODUR_USER*: Username for Lodur login
* *LODUR_PASSWORD*: Password of Lodur user
* *LODUR_BASE_URL*: Lodur base URL
* *HEARTBEAT_URL*: URL to send Hearbeat to (https://hchk.io/)
Environment variables can also be stored in a `.env` file.
## TODO
### Version 1
* Much more error handling
* Parse PDF
* Store parsed data in Lodur text fields for copy/paste
* Cleanup code into proper functions and classes
* Lodur "API" class
* Proper exit
* Healthchecks for Kubernetes probes
Before version 1 can be tagged, it must have processed at least 5 real
Before version 1 can be tagged, it must have processed at least 10 real
Einsätze!
### Known instabilities
* Storing files with "current year" doesn't work well during end of year
### Future versions
* Generalize
* Documentation
* IMAP idle
* IMAP Idle
* Display PDF on Dashboard
* Send statistics to InfluxDB
* Webapp to see what's going on
* Get as many data out of the PDFs as possible
* Simple webform to fill-in missing data (skipping Lodur completely)
* Webapp for chosing who was there during the Einsatz (tablet ready)
## WCPGW
What could possibly go wrong? A lot!
* Storing files with "current year" doesn't work well during end of year
* PDF layout may change without information
* PDF parsing may fail due to PDF creation instabilities
* Lodur forms can change without notice
* Error handling doesn't catch all cases
## Lodur Information Gathering
### eins_stat_kantone

19
emailhandling.py → library/emailhandling.py
View File

@ -16,7 +16,7 @@ class EmailHandling:
def __init__(self, server, username, password, mailbox, tmp_dir):
self.logger = logging.getLogger(__name__)
self.logger.info('Connecting to IMAP server ' + server)
self.logger.info('Connecting to IMAP server %s', server)
self.tmp_dir = tmp_dir
try:
@ -24,7 +24,7 @@ class EmailHandling:
self.imap.login(username, password)
self.imap.select(mailbox, readonly=False)
except Exception as err:
self.logger.error('IMAP connection failed - exiting: ' + str(err))
self.logger.error('IMAP connection failed - exiting: %s', str(err))
raise SystemExit(1)
self.logger.info('IMAP connection successfull')
@ -42,7 +42,7 @@ class EmailHandling:
return False
num_messages = len(msg_ids[0].split())
self.logger.info('Found ' + str(num_messages) + ' matching messages')
self.logger.info('Found %s matching messages', str(num_messages))
return num_messages, msg_ids
@ -63,14 +63,17 @@ class EmailHandling:
if isinstance(response_part, tuple):
mail = email.message_from_string(str(response_part[1], 'utf-8'))
subject = mail['subject']
self.logger.info('Getting attachment from: ' + subject)
f_type, f_id = self.parse_subject(subject)
self.logger.info('[%s] Getting attachment from "%s"', f_id, subject)
for part in mail.walk():
file_name = part.get_filename()
if not file_name:
self.logger.debug('Most probably not an attachment as no filename found')
self.logger.debug(
'Most probably not an attachment as no filename found'
)
continue
self.logger.info('Extracting attachment: ' + file_name)
self.logger.info('[%s] Extracting attachment "%s"', f_id, file_name)
if bool(file_name):
f_type, _ = self.parse_subject(subject)
@ -78,7 +81,7 @@ class EmailHandling:
# save attachment to filesystem
file_path = os.path.join(self.tmp_dir, renamed_file_name)
self.logger.info('Saving attachment to ' + file_path)
self.logger.info('[%s] Saving attachment to "%s"', f_id, file_path)
if not os.path.isfile(file_path):
file = open(file_path, 'wb')
file.write(part.get_payload(decode=True))
@ -87,7 +90,7 @@ class EmailHandling:
data[subject] = renamed_file_name
# mark as seen
self.logger.info('Marking message as seen ' + subject)
self.logger.info('[%s] Marking message "%s" as seen', f_id, subject)
self.imap.store(msg_id, '+FLAGS', '(\\Seen)')
return data

231
library/lodur.py Normal file
View File

@ -0,0 +1,231 @@
#!/usr/bin/env python3
""" Small Lodur Library for the Module 36 - Einsatzrapport """
import re
import logging
from datetime import datetime
import mechanicalsoup
class Lodur:
""" Lodur """
def __init__(self, url, username, password):
self.logger = logging.getLogger(__name__)
self.logger.info('Connecting to Lodur')
self.url = url
# MechanicalSoup initialization and login to Lodur
self.browser = mechanicalsoup.StatefulBrowser()
# The login form is located in module number 9
self.browser.open(self.url + '?modul=9')
self.browser.select_form()
self.browser['login_member_name'] = username
self.browser['login_member_pwd'] = password
self.browser.submit_selected()
# Check if login succeeded by finding the img with
# alt text LOGOUT
page = self.browser.get_current_page()
if page.find(alt='LOGOUT'):
self.logger.info('Login to Lodur succeeded')
else:
self.logger.fatal('Login to Lodur failed - exiting')
raise SystemExit(1)
def einsatzprotokoll(self, f_id, pdf_data, webdav_client):
""" Prepare Einsatzprotokoll to be sent to Lodur """
# check if data is already sent to lodur - data contains lodur_id
lodur_data = webdav_client.get_lodur_data(f_id)
if lodur_data:
# einsatz available in Lodur - updating existing entry
self.logger.info('[%s] Lodur data found - updating entry', f_id)
# when PDF parsing fails, pdf_data is false. fill with tbd when this happens
if pdf_data:
zh_fw_ausg = datetime.strptime(
pdf_data['ausgerueckt'],
'%H:%M',
)
zh_am_schad = datetime.strptime(
pdf_data['anort'],
'%H:%M',
)
else:
# Do nothing when no PDF data - we don't have anything to do then
self.logger.error('[%s] No PDF data found - filling in dummy data', f_id)
zh_fw_ausg = datetime.now()
zh_am_schad = datetime.now()
# Complement existing form data
self.logger.info('[%s] Preparing form data for Einsatzprotokoll', f_id)
lodur_data['zh_fw_ausg_h'] = zh_fw_ausg.hour # 13. FW ausgerückt
lodur_data['zh_fw_ausg_m'] = zh_fw_ausg.minute # 13. FW ausgerückt
lodur_data['zh_am_schad_h'] = zh_am_schad.hour # 14. Am Schadenplatz
lodur_data['zh_am_schad_m'] = zh_am_schad.minute # 14. Am Schadenplatz
# The following fields are currently unknown as PDF parsing is hard for these
#lodur_data['zh_fw_einge_h'] = UNKNOWN, # 15. FW eingerückt
#lodur_data['zh_fw_einge_m'] = 'UNKNOWN' # 15. FW eingerückt
#lodur_data['eins_erst_h'] = 'UNKNOWN' # 16. Einsatzbereitschaft erstellt
#lodur_data['eins_erst_m'] = 'UNKNOWN' # 16. Einsatzbereitschaft erstellt
# Submit the form
self.submit_form_einsatzrapport(lodur_data)
else:
# einsatz not available in Lodur
self.logger.error('[%s] No lodur_id found')
return False
def einsatzrapport(self, f_id, pdf_data, webdav_client):
""" Prepare form in module 36 - Einsatzrapport """
# when PDF parsing fails, pdf_data is false. fill with placeholder when this happens
if pdf_data:
date = datetime.strptime(
pdf_data['datum'],
'%d.%m.%Y',
)
time = datetime.strptime(
pdf_data['zeit'],
'%H:%M',
)
eins_ereig = pdf_data['einsatz']
bemerkungen = pdf_data['bemerkungen']
wer_ala = pdf_data['melder']
adr = pdf_data['strasse'] + ', ' + pdf_data['plzort']
else:
date = datetime.now()
time = datetime.now()
eins_ereig = 'UNKNOWN'
bemerkungen = 'UNKNOWN'
wer_ala = 'UNKNOWN'
adr = 'UNKNOWN'
# Fill in form data
self.logger.info('[%s] Preparing form data for Einsatzrapport', f_id)
lodur_data = {
'e_r_num': f_id, # 01. Einsatzrapportnummer
'eins_stat_kantone': '1', # 02. Einsatzart FKS
'emergency_concept_id': '2', # 03. Verrechnungsart
'ver_sart': 'ab', # 03. Verrechnungsart internal: ab, th, uh, ak, tt
'dtv_d': str(date.day), # 04. Datum von
'dtv_m': str(date.month), # 04. Datum von
'dtv_y': str(date.year), # 04. Datum von
'dtb_d': str(date.day), # 04. Datum bis - we dont know yet the end date
'dtb_m': str(date.month), # 04. Datum bis - assume the same day
'dtb_y': str(date.year), # 04. Datum bis
'ztv_h': str(time.hour), # 05. Zeit von
'ztv_m': str(time.minute), # 05. Zeit von
'ztb_h': str(time.hour + 1), # 05. Zeit bis - we dont know yet the end time
'ztb_m': str(time.minute), # 05. Zeit bis - just add 1 hour and correct later
'e_ort_1': '306', # 06. Einsatzort: Urdorf 306, Birmensdorf 298
'eins_ereig': eins_ereig, # 07. Ereignis
'adr': adr, # 08. Adresse
'wer_ala': wer_ala, # 10. Wer hat alarmiert
'zh_alarmierung_h': str(time.hour), # 12. Alarmierung
'zh_alarmierung_m': str(time.minute), # 12. Alarmierung
'ang_sit': 'TBD1', # 17. Angetroffene Situation
'mn': 'TBD2', # 19. Massnahmen
'bk': bemerkungen, # 20. Bemerkungen
'en_kr_feuwehr': '1', # 21. Einsatzkräfte
'ali_io': '1', # 24. Alarmierung
'kopie_gvz': '1', # 31. Kopie innert 10 Tagen an GVZ
'mannschaftd_einsa': '70', # 32. Einsatzleiter|in
}
# Submit the form
lodur_id, auto_num = self.submit_form_einsatzrapport(lodur_data)
# save lodur id and data to webdav
lodur_data['event_id'] = lodur_id
lodur_data['auto_num'] = auto_num
webdav_client.store_lodur_data(f_id, lodur_data)
return lodur_id
def einsatzrapport_alarmdepesche(self, f_id, file_path, webdav_client):
""" Upload a file to Alarmdepesche """
self.logger.info('[%s] Submitting Alarmdepesche to Lodur', f_id)
# check if data is already sent to lodur - data contains lodur_id
lodur_id = webdav_client.get_lodur_data(f_id)['event_id']
# Prepare the form
self.browser.open(self.url + '?modul=36&what=828&event=' + lodur_id)
self.browser.select_form('#frm_alarmdepesche')
# Fill in form data
self.browser['alarmdepesche'] = open(file_path, 'rb')
# Submit the form
self.browser.submit_selected()
self.logger.info('[%s] Alarmdepesche submitted', f_id)
def submit_form_einsatzrapport(self, lodur_data):
""" Form in module 36 - Einsatzrapport """
# Prepare the form
if 'event_id' in lodur_data:
# existing entry to update
self.logger.info(
'[%s] Updating existing entry with ID %s',
lodur_data['e_r_num'],
lodur_data['event_id'],
)
self.browser.open(
self.url +
'?modul=36&what=144&edit=1&event=' +
lodur_data['event_id']
)
else:
self.logger.info('[%s] Creating new entry in Lodur', lodur_data['e_r_num'])
self.browser.open(
self.url +
'?modul=36'
)
self.browser.select_form('#einsatzrapport_main_form')
# Prepare the form data to be submitted
for key, value in lodur_data.items():
# Encode some of the fields so they are sent in correct format
# Encoding bk causes some troubles - therefore we skip that - but it
# would be good if it would be encoded as it can / will contain f.e.abs
# Umlauts
self.logger.info('Form data: %s = %s', key, value)
if key in ('eins_ereig', 'adr', 'wer_ala'):
self.browser[key] = value.encode('iso-8859-1')
else:
self.browser[key] = value
# Submit the form
self.logger.info('[%s] Submitting form Einsatzrapport', lodur_data['e_r_num'])
response = self.browser.submit_selected()
self.logger.info('[%s] Form Einsatzrapport submitted', lodur_data['e_r_num'])
if 'event_id' in lodur_data:
return True
else:
# very ugly way to find the assigned event id by lodur
# lodur adds a script element at the bottom of the returned html
# with the location to reload the page - containing the assigned event id
lodur_id = re.search('modul=36&event=([0-9].*)&edit=1&what=144', response.text).group(1)
self.logger.info('[%s] Lodur assigned the event_id %s', lodur_data['e_r_num'], lodur_id)
# The hidden field auto_num is also needed for updating the form
# and it's written somewhere in javascript code - but not on the page
# delivered after the submission which contains the redirect URL
# It's only delivered in the next page. So we browse to this page now
content = self.browser.open(
self.url +
'?modul=36&edit=1&what=144&event=' + lodur_id
).text
auto_num = re.search(r"fdata\['auto_num'\]\[2\]='(.*)';", content).group(1)
self.logger.info('[%s] Lodur assigned the auto_num %s', lodur_data['e_r_num'], auto_num)
return lodur_id, auto_num

57
library/mqtt.py Normal file
View File

@ -0,0 +1,57 @@
#!/usr/bin/env python3
""" MQTT Functions """
import logging
import paho.mqtt.client as mqtt
class MQTTClient:
""" MQTT Client """
def __init__(self, server, username, password, base_topic):
self.logger = logging.getLogger(__name__)
self.logger.info('Connecting to MQTT broker %s', server)
try:
self.mqtt_client = mqtt.Client('pylokid')
self.mqtt_client.username_pw_set(username, password=password)
self.mqtt_client.tls_set()
self.mqtt_client.connect(server, 8883, 60)
self.mqtt_client.loop_start()
except Exception as err:
self.logger.error('MQTT connection failed - exiting: %s', str(err))
raise SystemExit(1)
self.logger.info('MQTT connection successfull')
self.base_topic = base_topic
def send_message(self, f_type, f_id, pdf_data=None, pdf_file=None):
""" Publish a message over MQTT """
topic = "{0}/{1}/".format(self.base_topic, f_id)
self.logger.info('[%s] Publishing information on MQTT topic %s*', f_id, topic)
if f_type == 'Einsatzausdruck_FW':
try:
self.mqtt_client.publish(topic + 'typ', 'Einsatzauftrag')
self.mqtt_client.publish(topic + 'einsatz', pdf_data['einsatz'])
self.mqtt_client.publish(
topic + 'datumzeit',
pdf_data['datum'] + ' - ' + pdf_data['zeit']
)
self.mqtt_client.publish(topic + 'sondersignal', pdf_data['sondersignal'])
self.mqtt_client.publish(
topic + 'adresse',
pdf_data['strasse'] + ', ' + pdf_data['plzort']
)
self.mqtt_client.publish(topic + 'hinweis', pdf_data['hinweis'])
self.mqtt_client.publish(topic + 'bemerkungen', pdf_data['bemerkungen'])
# Publish the PDF blob
pdf_fh = open(pdf_file, 'rb')
pdf_binary = pdf_fh.read()
self.mqtt_client.publish(topic + 'pdf', bytes(pdf_binary))
except IndexError as err:
self.logger.info('[%s] Cannot publish information: %s', f_id, err)
elif f_type == 'Einsatzprotokoll':
self.mqtt_client.publish(topic + 'typ', 'Einsatzprotokoll')

23
pdf_extract.py → library/pdf_extract.py
View File

@ -64,18 +64,18 @@ class PDFHandling:
# sanity check to see if we can correlate the f_id
if f_id == splited[14]:
self.logger.info('PDF parsing: f_id matches line 14')
self.logger.info('[%s] ID matches in PDF', f_id)
else:
self.logger.error('PDF parsing: f_id doesn\'t match line 14')
self.logger.error('[%s] ID does not match in PDF', f_id)
return False
try:
# search some well-known words for later positional computation
# search some well-known words for later positional computation
try:
index_bemerkungen = splited.index('Bemerkungen')
index_dispo = splited.index('Disponierte Einheiten')
index_hinweis = splited.index('Hinweis')
except:
self.logger.error('PDF file doesn\'t look like a Einsatzausdruck')
except IndexError:
self.logger.error('[%s] PDF file does not look like a Einsatzausdruck', f_id)
return False
# get length of bemerkungen field
@ -87,14 +87,15 @@ class PDFHandling:
'auftrag': splited[14],
'datum': splited[15],
'zeit': splited[16],
'melder': self.concatenate_to_multiline_string(splited, 18, 19),
'melder': splited[18] + ' ' + splited[19],
'erfasser': splited[20],
'bemerkungen': self.concatenate_to_multiline_string(
splited,
index_bemerkungen,
index_bemerkungen + 1,
index_bemerkungen + length_bemerkungen
),
).rstrip(),
'einsatz': splited[index_dispo+5],
'sondersignal': splited[index_dispo+6],
'plzort': splited[index_dispo+8].title(),
'strasse': splited[index_dispo+9].title(),
#'objekt': splited[],
@ -109,9 +110,9 @@ class PDFHandling:
# sanity check to see if we can correlate the f_id
if f_id == splited[26]:
self.logger.info('PDF parsing: f_id matches line 26')
self.logger.info('[%s] ID matches in PDF', f_id)
else:
self.logger.error('PDF parsing: f_id doesn\'t match line 26')
self.logger.error('[%s] ID does not match in PDF', f_id)
return False
data = {

58
webdav.py → library/webdav.py
View File

@ -3,6 +3,7 @@
""" WebDav Functions """
import os
import json
from datetime import datetime
import logging
import asyncio
@ -13,7 +14,7 @@ class WebDav:
def __init__(self, url, username, password, webdav_basedir, tmp_dir):
self.logger = logging.getLogger(__name__)
self.logger.info('Connecting to WebDAV server: ' + url)
self.logger.info('Connecting to WebDAV server %s', url)
self.loop = asyncio.get_event_loop()
self.webdav_basedir = webdav_basedir
@ -25,25 +26,25 @@ class WebDav:
password=password,
)
except:
self.logger.error('WebDav connection failed - exiting')
self.logger.error('WebDAV connection failed - exiting')
self.logger.info('WebDav connection successfull')
self.logger.info('WebDAV connection successfull')
def upload(self, file_name, f_id):
""" uploads a file to webdav - checks for existence before doing so """
# upload with webdav
remote_upload_dir = self.webdav_basedir + "/" + str(datetime.now().year) + "/" + f_id
self.logger.info('Uploading file to WebDAV:' + remote_upload_dir)
self.logger.info('[%s] Uploading file to WebDAV "%s"', f_id, remote_upload_dir)
# create directory if not yet there
if not self.loop.run_until_complete(self.webdav.exists(remote_upload_dir)):
self.logger.info('Creating directory ' + remote_upload_dir)
self.logger.info('[%s] Creating directory "%s"', f_id, remote_upload_dir)
self.loop.run_until_complete(self.webdav.mkdir(remote_upload_dir))
remote_file_path = remote_upload_dir + "/" + file_name
if self.loop.run_until_complete(self.webdav.exists(remote_file_path)):
self.logger.info('File ' + file_name + ' already exists on webdav')
self.logger.info('[%s] File "%s" already exists on WebDAV', f_id, file_name)
else:
self.loop.run_until_complete(
self.webdav.upload(
@ -51,53 +52,52 @@ class WebDav:
remote_file_path,
)
)
self.logger.info('File ' + file_name + ' uploaded')
self.logger.info('[%s] File "%s" uploaded', f_id, file_name)
def einsatz_exists(self, f_id):
""" check if an einsatz is already created """
remote_upload_dir = self.webdav_basedir + "/" + str(datetime.now().year) + "/" + f_id
if self.loop.run_until_complete(self.webdav.exists(remote_upload_dir)):
self.logger.info('Einsatz exists ' + f_id)
self.logger.info('[%s] Einsatz exists on WebDAV', f_id)
return True
else:
return False
def store_lodur_id(self, lodur_id, f_id):
""" stores assigned lodur_id on webdav """
def store_lodur_data(self, f_id, lodur_data):
""" stores lodur data on webdav """
file_name = f_id + '_lodurid.txt'
file_name = f_id + '_lodur.json'
file_path = os.path.join(self.tmp_dir, file_name)
if not os.path.isfile(file_path):
file = open(file_path, 'w')
file.write(str(lodur_id))
file.close()
self.logger.info('Stored Lodur ID locally in: ' + file_path)
self.upload(file_name, f_id)
else:
self.logger.info('Lodur ID already available locally in: ' + file_path)
def get_lodur_id(self, f_id):
""" gets lodur_id if it exists """
file = open(file_path, 'w')
file.write(json.dumps(lodur_data))
file.close()
file_name = f_id + '_lodurid.txt'
self.logger.info('[%s] Stored Lodur data locally in %s', f_id, file_path)
self.upload(file_name, f_id)
def get_lodur_data(self, f_id):
""" gets lodur data if it exists """
file_name = f_id + '_lodur.json'
file_path = os.path.join(self.tmp_dir, file_name)
# first check if we already have it locally - then check on webdav
if os.path.isfile(file_path):
with open(file_path, 'r') as content:
lodur_id = content.read()
self.logger.info('Found Lodur ID for ' + f_id + ' locally: ' + lodur_id)
return lodur_id
lodur_data = json.loads(content.read())
self.logger.info('[%s] Found Lodur data locally', f_id)
return lodur_data
else:
remote_upload_dir = self.webdav_basedir + "/" + str(datetime.now().year) + "/" + f_id
remote_file_path = remote_upload_dir + '/' + file_name
if self.loop.run_until_complete(self.webdav.exists(remote_file_path)):
self.loop.run_until_complete(self.webdav.download(remote_file_path, file_path))
with open(file_path, 'r') as content:
lodur_id = content.read()
self.logger.info('Found Lodur ID for ' + f_id + ' on WebDAV: ' + lodur_id)
return lodur_id
lodur_data = json.loads(content.read())
self.logger.info('[%s] Found Lodur data on WebDAV', f_id)
return lodur_data
else:
self.logger.info('No Lodur ID found for ' + f_id)
self.logger.info('[%s] No existing Lodur data found', f_id)
return False

149
lodur.py
View File

@ -1,149 +0,0 @@
#!/usr/bin/env python3
""" Small Lodur Library for the Module 36 - Einsatzrapport """
import re
import logging
from datetime import datetime
import requests
class Lodur:
""" Lodur """
def __init__(self, url, username, password):
self.logger = logging.getLogger(__name__)
self.session = requests.session()
# Authenticate
self.logger.info('Connecting to Lodur')
answer = self.session.post(
url,
data={
'login_member_name': username,
'login_member_pwd': password,
}
)
# When login failed the form has the CSS class error
# This is the only way to tell if it worked or not
if re.search('"login error"', answer.text):
self.logger.fatal('Login to Lodur failed - exiting')
raise SystemExit(1)
else:
self.logger.info('Login to Lodur succeeded')
def create_einsatzrapport(self, f_id, pdf_data):
""" Create a new Einsatzrapport """
params = (
('modul', '36'),
('what', '144'),
('sp', '1'),
('event', ''),
('edit', ''),
('is_herznotfall', ''),
)
# when PDF parsing fails, pdf_data is false. fill with tbd when this happens
if pdf_data:
date = datetime.strptime(
pdf_data['datum'],
'%d.%m.%Y',
)
time = datetime.strptime(
pdf_data['zeit'],
'%H:%M',
)
eins_ereig = pdf_data['einsatz']
adr = pdf_data['strasse'] + ', ' + pdf_data['plzort']
else:
date = datetime.now()
eins_ereig = 'TBD'
adr = 'TBD'
data = {
'e_r_num': (None, f_id), # 01. Einsatzrapportnummer
'eins_stat_kantone': (None, '1'), # 02. Einsatzart FKS
'emergency_concept_id': (None, '2'), # 03. Verrechnungsart
'ver_sart': (None, 'ab'), # 03. Verrechnungsart internal: ab, th, uh, ak, tt
'dtv_d': (None, str(date.day)), # 04. Datum von
'dtv_m': (None, str(date.month)), # 04. Datum von
'dtv_y': (None, str(date.year)), # 04. Datum von
'dtb_d': (None, str(date.day)), # 04. Datum bis - we dont know yet the end date
'dtb_m': (None, str(date.month)), # 04. Datum bis - assume the same day
'dtb_y': (None, str(date.year)), # 04. Datum bis
'ztv_h': (None, str(time.hour)), # 05. Zeit von
'ztv_m': (None, str(time.minute)), # 05. Zeit von
'ztb_h': (None, str(time.hour + 1)), # 05. Zeit bis - we dont know yet the end time
'ztb_m': (None, str(time.minute)), # 05. Zeit bis - just add one hour and correct later
'e_ort_1': (None, '306'), # 06. Einsatzort: Urdorf 306, Birmensdorf 298
'eins_ereig': (None, eins_ereig.encode('iso-8859-1')), # 07. Ereignis
'adr': (None, adr.encode('iso-8859-1')), # 08. Adresse
#'zh_alarmierung_h': (None, 'UNKNOWN'), # 12. Alarmierung
#'zh_alarmierung_m': (None, 'UNKNOWN'), # 12. Alarmierung
#'zh_fw_ausg_h': (None, 'UNKNOWN'), # 13. FW ausgerückt
#'zh_fw_ausg_m': (None, 'UNKNOWN'), # 13. FW ausgerückt
#'zh_am_schad_h': (None, 'UNKNOWN'), # 14. Am Schadenplatz
#'zh_am_schad_m': (None, 'UNKNOWN'), # 14. Am Schadenplatz
#'zh_fw_einge_h': (None, 'UNKNOWN'), # 15. FW eingerückt
#'zh_fw_einge_m': (None, 'UNKNOWN'), # 15. FW eingerückt
#'eins_erst_h': (None, 'UNKNOWN'), # 16. Einsatzbereitschaft erstellt
#'eins_erst_m': (None, 'UNKNOWN'), # 16. Einsatzbereitschaft erstellt
'ang_sit': (None, 'TBD1'), # 17. Angetroffene Situation
'mn': (None, 'TBD2'), # 19. Massnahmen
'bk': (None, 'TBD3'), # 20. Bemerkungen
'en_kr_feuwehr': (None, '1'), # 21. Einsatzkräfte
'ali_io': (None, '1'), # 24. Alarmierung
'kopie_gvz': (None, '1'), # 31. Kopie innert 10 Tagen an GVZ
'mannschaftd_einsa': (None, '70'), # 32. Einsatzleiter|in
}
# post data to create new einsatzrapport
answer = self.session.post(
'https://lodur-zh.ch/urdorf/index.php',
params=params,
files=data,
)
# very ugly way to find the assigned event id by lodur
# lodur really adds a script element at the bottom of the returned html
# with the location to reload the page - containing the assigned event id
lodur_id = re.search('modul=36&event=([0-9].*)&edit=1&what=144', answer.text).group(1)
return lodur_id
def upload_alarmdepesche(self, lodur_id, file_path):
""" Upload a file to Alarmdepesche """
params = (
('modul', '36'),
('what', '828'),
('event', lodur_id),
)
data = {
'alarmdepesche': open(file_path, 'rb')
}
self.session.post(
'https://lodur-zh.ch/urdorf/index.php',
params=params,
files=data,
)
# TODO this doesnt work. We first have to fetch the current form with its
# data, update the fields we want to change and resubmit the form
def update_einsatzrapport(self, lodur_id, data):
""" Update the Einsatzrapport """
params = (
('modul', '36'),
('what', '144'),
('sp', '1'),
('event', lodur_id),
('edit', '1'),
('is_herznotfall', ''),
)
answer = self.session.post(
'https://lodur-zh.ch/urdorf/index.php',
params=params,
files=data,
)
print(answer.headers)

108
main.py
View File

@ -10,14 +10,11 @@ import requests
from dotenv import find_dotenv, load_dotenv
# local classes
from emailhandling import EmailHandling
from lodur import Lodur
from mqtt import MQTTClient
from pdf_extract import PDFHandling
from webdav import WebDav
# TODO replace by IMAP idle
_INTERVAL = 10
from library.emailhandling import EmailHandling
from library.lodur import Lodur
from library.mqtt import MQTTClient
from library.pdf_extract import PDFHandling
from library.webdav import WebDav
# Configuration
load_dotenv(find_dotenv())
@ -25,6 +22,7 @@ IMAP_SERVER = os.getenv("IMAP_SERVER")
IMAP_USERNAME = os.getenv("IMAP_USERNAME")
IMAP_PASSWORD = os.getenv("IMAP_PASSWORD")
IMAP_MAILBOX = os.getenv("IMAP_MAILBOX", "INBOX")
IMAP_CHECK_INTERVAL = os.getenv("IMAP_CHECK_INTERVAL", "10")
WEBDAV_URL = os.getenv("WEBDAV_URL")
WEBDAV_USERNAME = os.getenv("WEBDAV_USERNAME")
WEBDAV_PASSWORD = os.getenv("WEBDAV_PASSWORD")
@ -33,6 +31,7 @@ TMP_DIR = os.getenv("TMP_DIR", "/tmp")
MQTT_SERVER = os.getenv("MQTT_SERVER")
MQTT_USER = os.getenv("MQTT_USER")
MQTT_PASSWORD = os.getenv("MQTT_PASSWORD")
MQTT_BASE_TOPIC = os.getenv("MQTT_BASE_TOPIC", "pylokid")
LODUR_USER = os.getenv("LODUR_USER")
LODUR_PASSWORD = os.getenv("LODUR_PASSWORD")
LODUR_BASE_URL = os.getenv("LODUR_BASE_URL")
@ -78,11 +77,13 @@ def main():
MQTT_SERVER,
MQTT_USER,
MQTT_PASSWORD,
MQTT_BASE_TOPIC,
)
# Initialize PDF Parser
pdf = PDFHandling()
# Main Loop
while True:
attachments = {}
num_messages, msg_ids = imap_client.search_emails()
@ -97,73 +98,84 @@ def main():
# Take actions - depending on the type
if f_type == 'Einsatzausdruck_FW':
lodur_id = webdav_client.get_lodur_id(f_id)
if lodur_id:
logger.info(
'Einsatzrapport ' + f_id + ' already created in Lodur: ' + lodur_id
)
# Upload Alarmdepesche as it could contain more information than the first one
lodur_client.upload_alarmdepesche(
lodur_id,
os.path.join(TMP_DIR, file_name),
)
else:
# this is real - publish Einsatz on MQTT
# TODO publish more information about the einsatz - coming from the PDF
mqtt_client.send_message(f_type, f_id)
logger.info('[%s] Processing type %s', f_id, f_type)
lodur_data = webdav_client.get_lodur_data(f_id)
# get as many information from PDF as possible
pdf_data = pdf.extract_einsatzausdruck(
if lodur_data:
logger.info(
'[%s] Einsatzrapport already created in Lodur', f_id
)
# Upload Alarmdepesche as it could contain more information
# than the first one
lodur_client.einsatzrapport_alarmdepesche(
f_id,
os.path.join(TMP_DIR, file_name),
webdav_client,
)
else:
## Here we get the initial Einsatzauftrag - Time to run
# get as many information from PDF as possible
pdf_file = os.path.join(TMP_DIR, file_name)
pdf_data = pdf.extract_einsatzausdruck(
pdf_file,
f_id,
)
# publish Einsatz on MQTT
mqtt_client.send_message(f_type, f_id, pdf_data, pdf_file)
# create new Einsatzrapport in Lodur
logger.info('Creating Einsatzrapport in Lodur for ' + f_id)
lodur_id = lodur_client.create_einsatzrapport(
lodur_client.einsatzrapport(
f_id,
pdf_data,
webdav_client,
)
logger.info('Sent data to Lodur. Assigned Lodur ID: ' + lodur_id)
# store lodur id in webdav
webdav_client.store_lodur_id(lodur_id, f_id)
logger.info(
'Uploading PDF for ' + f_id + ' to Lodur Einsatzrapport ' + lodur_id
)
lodur_client.upload_alarmdepesche(
lodur_id,
# upload Alarmdepesche PDF to Lodur
lodur_client.einsatzrapport_alarmdepesche(
f_id,
os.path.join(TMP_DIR, file_name),
webdav_client,
)
elif f_type == 'Einsatzprotokoll':
logger.info('[%s] Processing type %s', f_id, f_type)
# Einsatz finished - publish on MQTT
mqtt_client.send_message(f_type, f_id)
mqtt_client.send_message(f_type, f_id, pdf_data, pdf_file)
lodur_id = webdav_client.get_lodur_id(f_id)
if lodur_id:
logger.info('Uploading Einsatzprotokoll to Lodur')
lodur_client.upload_alarmdepesche(
lodur_id,
lodur_data = webdav_client.get_lodur_data(f_id)
if lodur_data:
# Upload Einsatzprotokoll to Lodur
lodur_client.einsatzrapport_alarmdepesche(
f_id,
os.path.join(TMP_DIR, file_name),
webdav_client,
)
# Parse the Einsatzprotokoll PDF
pdf_data = pdf.extract_einsatzprotokoll(
os.path.join(TMP_DIR, file_name),
f_id,
)
# only update when parsing was successfull
if pdf_data:
logger.info('Updating Einsatzrapport with data from PDF - not yet implemented')
else:
logger.info('Updating Einsatzrapport not possible - PDF parsing failed')
# Update entry in Lodur with parse PDF data
lodur_client.einsatzprotokoll(f_id, pdf_data, webdav_client)
else:
logger.error('Cannot process Einsatzprotokoll as there is no Lodur ID')
logger.error(
'[%s] Cannot process Einsatzprotokoll as there is no Lodur ID',
f_id
)
else:
logger.error('Unknown type: ' + f_type)
logger.error('[%s] Unknown type: %s', f_id, f_type)
# send heartbeat
requests.get(HEARTBEAT_URL)
# repeat every
time.sleep(_INTERVAL)
logger.info('Waiting %s seconds until next check', IMAP_CHECK_INTERVAL)
time.sleep(int(IMAP_CHECK_INTERVAL))
if __name__ == '__main__':
try:

29
mqtt.py
View File

@ -1,29 +0,0 @@
#!/usr/bin/env python3
""" MQTT Functions """
import logging
import paho.mqtt.client as mqtt
class MQTTClient:
""" MQTT Client """
def __init__(self, server, username, password):
self.logger = logging.getLogger(__name__)
self.logger.info('Connecting to MQTT broker ' + server)
try:
self.mqtt_client = mqtt.Client('pylokid')
self.mqtt_client.username_pw_set(username, password=password)
self.mqtt_client.tls_set()
self.mqtt_client.connect(server, 8883, 60)
self.mqtt_client.loop_start()
except Exception as err:
self.logger.error('MQTT connection failed - exiting: ' + str(err))
raise SystemExit(1)
self.logger.info('MQTT connection successfull')
def send_message(self, f_type, f_id):
""" Publish a message over MQTT """
self.mqtt_client.publish('pylokid/' + f_type, f_id)

3
requirements.txt
View File

@ -1,5 +1,6 @@
aioeasywebdav==2.4.0
MechanicalSoup==0.9.0.post4
paho-mqtt==1.3.1
pdfminer.six==20170720
python-dotenv==0.7.1
requests==2.18.4
requests==2.18.4