Good morning everyone,
A feature I would love to have is an automatic notification in case WebGrab+ failed to grabbed the EPG information. The program works well enough that I can just have it to run automatically and not care about it, but once every 5-6 months something goes wrong (either the website from which it is grabbing changed something on their side, or donation license expires, etc.), and when that happens I only notice it when my TV guide is suddenly empty, which mean failed recordings, etc.
So a notification system (such as an automatic email, or otherwise) to let you know that one channel failed to update the EPG would be nice. That way we could fix the issue proactively rather than reactively.
Thank you and have a great day.
+1 on this request!
I would add a few additional requests to extend this one. A provider may remove a specific channel from their listing. This will result in an error message when trying to pull info for that channel. A failure notification can potentially include a list of specific channels that failed EPG retrieval.
At the very least, can you please include these error messages at the bottom summary section of the output log file? This would make it easier to grab it and process it with a simple script.
The idea is not bad, but...there are too many factors that could lead down to fill up your mail box with notification.
1. a mail system should be installed (windows, linux, android, etc.etc.)
2. A filtering criteria should be implemented. Here comes the worst part, you could have set timeout too low. A site could be temporarly down, but tomorrow is ok or many other factors, that something goes wrong.....a simple channel site_id code change and you do not get epg unless you run a fresh list.
I will check with authors if they want to implement that, may be with a cheklist for mail notification.
I have played with that idea over the years and came to the conclusion that it was not feasible in a reliable way. Many reasons, some of which given above by mat. However, something local (user side), could be made as a postprocess addon. For example a small program that compares the size of the created xmltv file with that of the previous day. If the difference is above a certain value it could issue some warning ..
Problem remains, what kind of warning ?? That's rather difficult, it has to be something the user alerts, even it wg++ run on another computer of the home network . Or maybe even alerting something on his mobile ?
Someone with a bright idea ???
here's my kick at it..
using linux and sed
i setup 2 channels in my config to fail
one to fail with "no shows in index page"
one to fail with "no index page recieved"
created a sh file with the below..
put it in the same directory as your webgrab log file and run it.
#!/bin/sh
sed -n '/\[Error \] no shows in indexpage/{x;p;d;}; x' WebGrab++.log.txt > NoShows.txt
sed -n '/\[Error \] no index page data received.*/p' WebGrab++.log.txt > NoIndex.txt
exit
Thanks BlackBear199, i think most user would like a way to run kind of an alarm (more likely via an email) that advice you of the problem. If i open my log and a search for no index or no show or exceed timeout i can find the problem, more likely as i said above, a mail system should made available or integrated.
Time ago i tried a light mail system on linux with some scripts.....i found more "easy" to search directly in the log once on a while for errors (same as your sed's). May be now we could integrate some alarms via email, like your license will expire in a week...you have 2 channels with no index page....or 5 channels with timeout, again we fall into a mail service for the 3 different platform.
Here's a "phase 1" suggestion:
at the bottom of the WG++log file, have a proper summery of all errors/and alerts in a few lines, just as you have the bottom summary of number of shows, total, time and date.
If you have them in a few summary lines, I am happy to contribute my time to write a post processing script that will ingest all this data and also send out an email to the user.
BTW, here is an example of the Gmail API, where every user can create his own API and you guys can provide the pipes in your app to connect to it.
https://mailtrap.io/blog/send-emails-with-gmail-api/
Here is an example of a Windows PowerShell script that reads the last line of the log file, extracts the finish and total times and creates a json file output.
# Get the contents of the file and split it into an array of lines
$fileContents = Get-Content -Path "WebGrab.log__0.txt"
$lines = $fileContents -split "\r?\n"
# Get the last line of the file and extract the finished time and total time
$lastLine = $lines[-1]
$finishedTime = $lastLine -replace ".*finished at ([0-9]{2}/[0-9]{2}/[0-9]{4} [0-9]{2}:[0-9]{2}:[0-9]{2}) done in .*", '$1'
$totalTime = $lastLine -replace ".*done in (.*)", '$1'
# Create a PowerShell object with the status information
$statusObject = @{
FinishedTime = $finishedTime
TotalTime = $totalTime
}
# Convert the PowerShell object to JSON
$statusJson = ConvertTo-Json $statusObject
# Write the JSON to a file
Set-Content -Path "status.json" -Value $statusJson
maybe were thinking too far outside the box.
why not use what we already have....webgrab itself.
here the ini if anyone want to mess with it.
its a pretty simple one,add the path(s) to your WebGrab++.log.txt file(s) on the subpage.format {list|} line.
i also changed it to only output something to the xml file if a problem is detected.
updated 16 mar 2023 revision 1
Thank you for the Analyzer file!
I am happy to try it out!
A couple of questions though. After updating the location of the wg log file, where does this file need to be saved at? How do we make sure it is triggered post grab completion? any post processing that needs to be added to the config?
On a separate note, I did create a small Python script that uses Sendgrid (which every person can create a free account for their email API, up to 1000 emails a month, so should be more than enough!) that scrapes the current log file (I can change/add the new input file from the analyzer) for the last row to get the process completion date & time as well as total time and send it to your email.
I can create a readme file as well with the install instructions for anyone interested, including the process of setting up your account on Sendgrid. could easily be added as a post process.
Here is the code, for now, and I attached the output email that I got from running it:
import sendgrid
import os
from sendgrid.helpers.mail import Mail, Email, To, Content
# Open the log file in read mode
with open('WebGrab++.log.txt', 'r') as f:
# Read the last line of the file
last_line = f.readlines()[-1]
# Extract the finish time and total time from the last line
finish_time = last_line.split()[-7] + ' ' + last_line.split()[-6]
total_time = last_line.split()[-3] + ' ' + last_line.split()[-2] + ' ' + last_line.split()[-1]
# Set up the email
sg = sendgrid.SendGridAPIClient(api_key=os.environ.get('SENDGRID_API_KEY')) # This is set as an OS environment variable
from_email = Email("THE SENDGRID VERIFIED EMAIL") # Change to your verified sender
to_email = To("YOUR EMAIL") # Change to your recipient
subject = "EPG creation status"
content = html_content=f'The task finished at {finish_time} and took {total_time} to complete.'
mail = Mail(from_email, to_email, subject, content)
# Get a JSON-ready representation of the Mail object
mail_json = mail.get()
# Send the email
response = sg.client.mail.send.post(request_body=mail_json)
print(response.status_code)
print(response.headers)
It would be preferable is such a Failure Notification is developed in the form of a PostProcess Plugin . http://webgrabplus.com/content/introducing-postprocess-plugin. I personally think that the trigger to issue the notification can be if the size of the xmltv differs to a certain extend wrt the previous one. The content of the notification message can be an extract of the latest log file as suggested above or just a simple .. have a look at your latest wglog files ..
what i do it run it with another instance of webgrab.
and as jan suggested use the post processor to run,i run rex after my grab has completed so i just add another line with grabbing disabled..
<postprocess grab="y" run="y">rex</postprocess>
<postprocess grab="n" run="y">/raiddata/0/NAS_WebGrab/analyzer/analyzer.sh</postprocess>
i created a folder named analyzer
inside that put a copy of webgrab config.xml and analyzer.ini
one channel for config...
<channel update="i" site="analyzer" site_id="" xmltv_id="Analyzer">Analyzer</channel>
you can run webgrab with custom paths by using
/path/to/webgrab/exe/dll /path/to/webgrab/config/folder
use the existing path to your webgrab exe or dll,you dont need another copy of the files.
my alanyzer.sh
#!/bin/bash
#3.1
mono /path/to/webgrab/exe/WebGrab+Plus.exe /path/to/webgrab/config/folder
V4.x/5.x
dotnet /path/to/webgrab/dll/WebGrab+Plus.dll /path/to/webgrab/config/folder
exit
use whatever line applies to you,not both
webgrab is installed on my nas in custom paths and mono is installed as a app in non standard locations so i specify full paths for everything
/raiddata/0/module/mono/sys/bin/mono /raiddata/0/NAS_WebGrab/bin/WebGrab+Plus.exe /raiddata/0/NAS_WebGrab/analyzer
thats it,analyzer will run after webgrab has completed.
btw nothing going to be bullet proof for catching all error as there many ways webgrab could fail.
the only true way is to have something added to the code itself.
updated the analyzer.ini in the post above.
made some improvements
1.title format => Channel: [channel-name]
2.subtitle format => SinteIni: [sitini-name]
3.description format => Problem(s) Detected: [problem-one], [problem-two], [ect]
*
values detected..
1.no index page data received
2.no shows in indexpage
3.suspicious title in index page
4.conflicting stoptime or duration
5.skipped show without a title
6.skipped : no value for next starttime
7.time parsing error : String was not recognized as a valid DateTime
another update.
i was contemplating adding the log start/stop/duration because if more than one show has a detected problem the same output would be added to every show.
i figured out a way to do it so if more than 1 show has a detected problem the log(s) start/end/duration info is only added to the first show.
also if your analyzing multiple log files,each log start/stop/duration will be listed.
Thank you for all the updates Blackbear199.
With your permission, would you mind explaining the scenarios with examples of what you are looking for as far as error types?
I would like to add those scenarios to the python script I created.
Main reasons for that:
- I can compile the Python script in a way that will accept env variables, so anyone can use it as a simple post processor within the same single grab run.
- The Python code supports sending emails with a summary notification and I can include an error report in the email.
- This post processor would support all OS platforms.
So again, if you would be so kind to include and explanation of the scenarios you are looking for, that would be great!
Happy to take the discussion offline if you'd like.
the errors are grabbing errors i can think of.
i wouldnt even try for things like "Unhandled exception",ect errors.
these are system errors not grabbing errors.
the errors that are listed are the ones that will give a hint as there may be something wrong with epg grabbing.
all of these point to something broken in the ini itself.
Thanks.
Since I am fairly new to WG+, I am not too familiar with what to look for in the examples you have provided.
If you can share a log entry example for each scenario, I can work with that. I can also add a check to validate size of EPG, before and after, for big variations.
values detected..
1.no index page data received
2.no shows in indexpage
3.suspicious title in index page
4.conflicting stoptime or duration
5.skipped show without a title
6.skipped : no value for next starttime
7.time parsing error : String was not recognized as a valid DateTime
search the forums for each error,thats what i did and used the log files that users uploaded as a reference.
its hard to manually duplicate some of them.
All versions of WG++ contain a statistic data collection. That data is currently not being used but was once used to upload to our site . We disabled it because it overloaded the statistic module that was supposed to analyse that data.
The statistic data collection however is still being done when for a certain config mode value.
That data contains the following components :
WG version
siteini_name and revision
Some values for channels, inactive and devel .. I don't remember what they meant .. will look into it if this data is usefull
OS version
And a list of errors :
- robots disallow
- cookie not found
- no indexpage
- no shows
- datatime error
- no title
- runtime error
I tried siteini tuamc.tv, rev 0 and inserted an error in the url_index :
data=[{"wgV1":5,"wgV2":0,"wgV3":1,"wgV4":2,"siteini_name":"tuamc.tv","siteini_version":0,"channels":1,"inactive":2,"devel":1,"OS":"Microsoft Windows NT 10.0.22621.0","errors":{"no_indexpage":1}}]
It issues such a data line for each siteini in the config .. example of two siteinis :
data=[{"wgV1":5,"wgV2":0,"wgV3":1,"wgV4":2,"siteini_name":"movistarplus.es","siteini_version":23,"channels":1,"inactive":1,"devel":1,"OS":"Microsoft Windows NT 10.0.22621.0","errors":{"no_shows":1}},{"wgV1":5,"wgV2":0,"wgV3":1,"wgV4":2,"siteini_name":"tuamc.tv","siteini_version":0,"channels":1,"inactive":0,"devel":1,"OS":"Microsoft Windows NT 10.0.22621.0"}]
The first, movistarplus.es had an non responding channel .. error no_shows
If this is of any use for the purpose of failure notification , it is rather easy to save that data into a file.
(it will need some 'repair' though .. it became a bit rusty)
i think that would be more than helpful jan.
If enabled .. it will issue an error in the log :
statistics upload error: The remote server returned an error: (404) Not Found.
That is because that function is not available anymore on the website .. but I don't think that is a problem .
Did a quick look at the details :
channels .. is the number of channels for a siteini that have a config entry with a update value . I needed to update the code , was broken
inactive .. is the number of channels for a siteini that resulted in no shows for whatever reason
devel .. is 1 if the development mode in the config is set .. (value not public !)
Hey guys,
Quick progress status on my end. I wanted to create a post process app that will take all the activities, including error notifications to the next level.
I have written a Python code app that is cross platform (should be... tested on Windows, but all the code was written to support both, needs testing) and currently has the following parameters as options for the post process, All parameters are optional with defaults assigned where relevant:
post-epg-process.exe github:y ghpathdepth:# epgtarget:path/to/folder gz:y email:email@example.com sizecheck:y onerror:stop
Command line arguments:
* sizecheck - options are y or n (default: n). Checks if the output XML EPG file differs by more than 1mb than the previous version (the app creates a json file to save previous size for comparison).
* gz - options are y or n (default: n). Would you like to compress the EPG file to gz format. Overwrites the gz file, does not delete the XML file.Saves in the same folder.
* epgtarget - provide a target path (default is: not defined, meaning no need to copy it). in case you want to copy the EPG file to a different location from where it is created. Will copy the xml or the gz format file, depending on if you chose to compress it.
* github - options are y or n (default: n). asked if you like to upload to GitHub at the end.
* ghpathdepth - provide a number (default: 0). In case you want file is copied into a sub folder of the GitHub cloned repo. Defined the folders depth from the repo root.
* email - email address (default is: not defined). If defined then will send the output of all the checks and process to your email. using sendgrid free service (example of a successful process completion email is attached as image). Whether or not you define the email, it will also save a log file locally with the same output.
* onerror - options are stop or continue (default is stop). Will either continue the process or stop on error to avoid uploading a defective EPG file.
It would be very easy to take it to the next level by parsing the WG+ log file and scanning for errors.
As I previously mentioned, I am not too familiar with the types of error messages to look for. If you can share just a sample log line and what to look for for in the line, I can add those checks to the tool. Currently, I am only looking for any line stating with "[Error" and presenting the full line.
I would gladly share with the community, obviously, once finalized and tested properly.
Looking forward for your insights!
you can use the regex expressions from my analyzer.ini
\[\s*(?:Error\|Debug)\s*\]\s*(no index page data received\|no shows in indexpage\|suspicious\s*title in index page\|conflicting stoptime or duration\|skipped show without a title\|skipped : no value for next starttime\|time parsing error :[^\n]*)
webgrab uses the | as a separator for multi value elements(arrays) so we need to escape these in webgrab code but any other language you probably dont use escapes..
\[\s*(?:Error\|Debug)\s*\]\s*(no index page data received|no shows in indexpage|suspicious\s*title in index page|conflicting stoptime or duration|skipped show without a title|skipped : no value for next starttime|time parsing error :[^\n]*)
OK, progress.
I had to parse the regex a bit differently in order for it to work properly with Python code. With that, I wanted to make sure the search terms can be updated periodically and autonomously from the app. That's why I have the search terms saved in a JSON file, and once this is added to the GitHub repo as a post process, I can add an automatic check to see if a new version exists to download a copy with updated terms.
These are the terms I am currently looking for:
-----------------------------------------------
{
"err_search_terms": [
"no index page data received",
"no shows in indexpage",
"suspicious\\s*title in index page",
"conflicting stoptime or duration",
"skipped show without a title",
"skipped : no value for next starttime",
"time parsing error :",
"add_logo\\.exe not found in"
]
}
BTW, side note, I added the last line because I noticed that it added it to my log file, even though I did not add it as a post process. I only added my EPG handler, and it added this error message (probably a bug!?). Either way, it is a good error message to show.
My ask, before I share with you guys for testing, can you share any large log files samples with different kinds of errors in it? I want to test it bit more.
I also added another argument: mode:run|test|setup.
default is 'run' - runs the process as is with all the optional arguments.
'test' - only checks the log file for the search terms and ignores any other operations.
'setup' - provide some automated setup processes to help configure external services (GitHub, Sendgrid).
Added an updated email image from my latest run.
best to search the forums for logs users uploaded.
some of these errors are very hard to produce manually.
you would have to create a epg file with info that would trigger the error and have webgrab read it.
Nice!
looking good
Currently only a Windows executable. Need to compile for Mac/Linux still.
I will compile for both as well, create a readme file and add it all to a zip file for downloading. would be interesting to get some community feedback :)
OK, I need some help here...
Everything is fully setup, but it seems that the post process option in the config file is not working well...
I am using v3.3 on Windows 11. The config post process section is in pic1.
The bottom output from the log file is in pic2
As you can see, a. I am not trying to run the add_logo post process app. b. it is not running my app at all. No log event of even being initiated is listed in my log file, since I am logging everything. Obviously, running the exact same command from command line with all the arguments works as expected.
Is the post process function broken?
try useing the full path to the exe file.
or put it in your webgrab config directory.
i just tried it with a exe with arguments,seems fine..
<postprocess grab="y" run="y">merge-xmltv.exe -t LOCAL -o X:\guide1.xml X:\guide.xml</postprocess>\>
part of log output..
[ Info ] Job (merge-xmltv.exe -t LOCAL -o guide1.xml guide.xml) started at 20/03/2023 02:21:30
[ Info ]
Merge-Xmltv V1.0.0
A Utility To Merge Multiple xml Files.
AND/OR Perform Time Corrections.
merge-xmltv.exe -h or --help For More Information.
Time Conversion Code Based on WG2MP.exe Source.
Courtesy of Jan van Straaten http://www.webgrabplus.com
Blackbear199 [12/03/2022]
Ttimezone: LOCAL
Output File:
X:\guide1.xml
Input Files:
X:\guide.xml
ect...
i then put the exe file somewhere other than the webgrab config directory and got the same error as you.
so use either the full path or put it in the webgab config directory.
seesm the error is triggered when the file cannot be found.
That's exactly it...
The exe file was in the the WG+ config folder... also, it does not explain why I get the add_logo error message.
Something is off. I can try to add the full path to the WG+config folder as well, but I doubt that is the issue, since it was already there.
the postprocessor was originally only designed to run one of 2 internal ones,mdb or rex.
people were asking for ways to manually add custom channel logo's and jan expanded this to use logo.exe which he wrote and is available in the postprocessor folder on git.
your no doubt seeing some hard coded error responses from this.
i dont think he thought of users using their own custom postprocessor at that time.
i was able to duplicate your error.
let me guess...your using dotnet webgrab version?
file in webgrab config directory.
V3.x without full path to file - works.
V4.x/5.x without full path to file - same error as you get about logo.exe
with full path to file - works.
file anywhere else - full path must be used.
- if i remember correctly the only exception to this is the siteini.pack and siteini.user folders and subdirectories.
this was added for a special function set(type=run) that can be called from ini to run external programs.
it allows the file to stored with the ini in its country folder for example without the need to specify the full path to it.
course,there seems to be a bug for dotnet version from what i said above.
for dotnet from the error its looking in the location of the ini that your webgrab config is using.
i got this error
add_logo.exe not found in /raiddata/0/NAS_WebGrab/IPTV/siteini.user/
thats the location of the ini's that my webgrab config uses.
Actually... no...
I am using the Mono version (v3.3).
The only reason I wrote it to be compatible with the post process option was specifically because Jan said that it would be preferable to be supported by it and designed for it :)
Obviously, using a simple batch file does the trick, running WG first and then this one.
Hi .. I just checked .. indeed the error message when the exe wasn't located by the plugin has a --stupid-- text --"add_logo.exe not found in .."
Can someone send me the exe to try ?? Use our wg++ mail .. webgrabpp@gmail.com
Sent to your email.
BTW, just had it run again. As you can see in the image, with the full path to the executable, I got a bit more details in the log file.
I was also able to watch the cli screen while it reached that point (unfortunately, did not manage to take a screenshot) and saw that the error message was related to a permission error.
Hope this helps.
your effort has inspired me to make mine attempt better,stop doing that!
j/k
wrote a php script to parse my report and display it on webgpage on my server.
NICE! love it :)
I had also made progress. As I started this journey I did not expect to go so deep into this rabbit hole... LOL
It is now more of a complete WG+Manager, if you'd like.
Current features:
- Full initial step by step setup manager, asking you questions and saving all results to a local json file so it can run without any CLI arguments.
- By default, running it without any parameters, it will only inspect the log file, and currently I also have it specifically state and aggregate the errors to provide a more meaningful error message. Love your PHP output page! I also added a similar option.
- gzip option for the file
- copy to a different path (like a NAS device)
- check significant file size changes
- Upload to GitHub. If GitHub is enabled, will also create an HTML page to see status summary (like your PHP :)
- Choose whether to stop or continue on major error detection
- Define scheduling for how often and when would you like to run the grab process - will create the relevant system scheduler tasks from it
- Define if you'd like the PC to reboot at an interval of some sort after a grab is complete (always good to prevent slowness)
- Added a process that pulls all the sites you are using in your config file, run Chanels creation for all of them, then compare against all the current channels in the config file and provide an differences output report (still need to finish this one and add to email/html). Very useful to identify early cases of channel changes in both ID or Name.
- Integrate with Sendgrid for email notifications.
- Kick-off and connect to VPN (NordVPN for now, can add support for more) connection to the page country for sites that need it.
- Auto-update process for the app and search terms from GitHub.
And more coming :)
Takes a while to get progress, as I am doing it in between my actual day job :D
i slowly catchin you.
i got php to send mail to my mail server,i added my email to outlook and i can see in the logs that its logging into but Xeams isnt forewarding it to outlook inbox and vice versa.email sent from outlook isnt making it to my Xeams mail server.
i can send a email from my hotmail account and see it arrive at my Xeams server though.
i think my ISP may be blocking port 25.
anyway,back to scratchin my head...
Edit:
works now.
Hey guys,
Been a while since I last posted any updates on this one...
I actually had some time to really spend some extended time on the tool and build it in a robust way to include many of the features I saw as useful for a person that both wants to automate the process, but also be notified by issues and easily solve them. Attached are a few of the recent views from the tool.
Current features:
- A full setup wizard that guides you through the setup process and validates your settings as you setup the system.
- WG+ application update notifications.
- Site.ini files that are being used are checked for online updates. If an update is found, it automatically downloads the latest version. It also detects usage of dummy sites, even if you renamed the file.
- License status and info, with license expiration notice.
- Full log analysis, including a web interface that parse the data and gives you a web interface into the WG+ log file to view the error directly.
- identify and notify when all channels from a site have errors, to flag the site with issues.
- Identify cases where a channel had a retry error but eventually succeeded to filter out from error results.
- The WG+ web interface of the log file gives you full search capabilities to simplify investigation.
- In the web interface, if you are using the add logo capabilities, it will pull the channel logos automatically.
- The ability to compress your output file to a .gz format.
- The ability to upload your EPG file to a GitHub repository post generation automatically.
- Sending out summary notifications are supported with: email, Telegram, Discord, Slack and WhatsApp.
- (in final development) A web report showing running a weekly compare of channel creation output compared to your existing WG+ conf file. This way you can see any mismatches in a simple way.
The tool has a batch file that manages running the different Python scripts. The batch file also validates that the requires Python support and needed libraries are installed and updated. It currently supports Windows platforms, as is what I have. With a bit of work and testing, it could support Linux as well, but will need help to test.
For now, if anyone is interested in testing, I am happy to have a few people test before releasing to everyone. The plan is to have it available via a GitHub repository. This way I can manage the utility itself to have an auto-update capability, whoever new enhancements and/or bug fixes are released.
Thoughts!?
A couple more images that did not upload...
nice,u must have put considerable time into this.
i'm sure there will be many that would have a use for it.
i'm not currently running any epg so have nothing to test it with.
I just completed the final remaining item over the weekend.
I now have a full report, updated weekly, that checks current config against WG+ updated channel creation output. The list then shows your existing channel list in your config for that site and their channel names, compared to your config channel names (just in case you changed them). I also included a table for channels in your config that no longer have a corresponding site_id channel in the site, and also a table to show you which channels are available on the site that are currently not in your config.
As much visibility as possible:
This looks helpful. Can't wait to see how it works.
Hi .. I haven't followed this topic for some time, so I don't know in what way the failure report grabs the attention of the user. I believe, the original idea was to send it by email. That, indeed, will alert the user when he checks his mail. A saved document someplace requires the same unwanted action as looking at the logfile or inspecting the xmltv file.
Recently I have done several experiments with sending emails automatically by means of a program. If the code for that is of any help I can provide that.
he has notifications working already.
- Sending out summary notifications are supported with: email, Telegram, Discord, Slack and WhatsApp.
Pages